Automatically improve data quality

Abstract

The method, system, and computer-readable medium include: receiving a request for a digital component from a user device; determining user attributes based on information provided by the user or information included in the request; identifying a behavioral model corresponding to the attributes; dynamically changing the presentation of an item depicted by the digital component based on the identified behavioral model; determining that the user corresponds to an underrepresented segment of a user group in a database containing information about the item; and, in response, generating a digital component including the dynamically changed presentation of the item; requesting feedback from the user about the item and including a feedback mechanism; updating the database to include the received feedback; and modifying the presentation of the item when it is distributed to other users with the user's attributes, at least in part based on the received feedback.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Application No. 16 / 942,937, filed July 30, 2020, and U.S. Provisional Application No. 63 / 010,399, filed April 15, 2020, the contents of which are incorporated herein by reference. Background Technology

[0003] This document relates to a data collection and model generation process that continuously reduces bias introduced by using data points from unevenly distributed or underrepresented populations and explores the design space. Summary of the Invention

[0004] Typically, an innovative aspect of the subject matter described in this specification can be embodied in a method for improving the robustness of a dataset executed by one or more data processing devices, the method comprising receiving from a user device a request for a digital component to be presented at the user device; determining one or more attributes of a user based on one or more of information provided by the user or information contained in the request for the digital component; identifying a behavioral model corresponding to the one or more attributes of the user; dynamically changing the presentation of an item depicted by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user; determining that the user has a specific attribute corresponding to an underrepresented segment of a user group in a database containing information about the item; in response to determining that the user has a specific attribute corresponding to an underrepresented segment of the user group; generating a digital component including the dynamically changing presentation of the item; requesting feedback from the user about the item and including a feedback mechanism enabling the user to submit feedback about the item; updating the database to include the feedback about the item obtained from the user; and modifying the presentation of the item when distributed to other users having one or more attributes of the user, at least in part based on the feedback obtained from the user.

[0005] These and other embodiments may each optionally include one or more of the following features.

[0006] In some implementations, the method includes, in response to receiving feedback from a user having specific attributes corresponding to an underrepresented segment of the user group, tagging the feedback information using one or more attributes of the user, and storing the tagged feedback information in a tagged, searchable database.

[0007] In some implementations, determining that a user has a specific attribute corresponding to an underrepresented segment of the user group in the database includes using statistical analysis to identify the underrepresented segment of the user group.

[0008] In some implementations, modifying the presentation of an item when it is distributed to other users who have one or more attributes of the user, based at least in part on feedback obtained from the user, includes selecting specific feedback mechanisms included in the digital components.

[0009] In some implementations, dynamically altering the presentation of items depicted by digital components based on an identified behavioral model corresponding to one or more attributes of the user includes using machine learning or artificial intelligence techniques to identify feedback to be requested from the user regarding the items.

[0010] In some implementations, determining one or more attributes of a user is based on information provided by the user, and dynamically changing the presentation of an item depicted by digital components based on an identified behavioral model corresponding to one or more of the user's attributes includes updating the identified behavioral model based on information provided by the user.

[0011] In some implementations, determining one or more attributes of a user is based on information contained in a request for a digital component, and dynamically changing the presentation of an item depicted by a digital component based on an identified behavioral model corresponding to one or more of the user's attributes includes updating the identified behavioral model based on information contained in the request for the digital component.

[0012] In some implementations, the method includes selecting the format of the feedback requested for a digital component that requests feedback from a user about a project, and verifying the information requested by the digital component based on specific attributes corresponding to an underrepresented segment of the user group.

[0013] In some implementations, the digital component displays the suit and two different product design shapes for the footwear, and feedback from the user about the item includes a selection of one of the two different product design shapes for the footwear that the user feels would match the suit better.

[0014] In some implementations, the digital component displays two different product design shapes of the car, the digital component specifies a particular subjective product design shape descriptor, the feedback mechanism is a slider, and the feedback obtained from the user about the item includes the user's choice of one of the two different product design shapes of the car that they believe can be better described by the particular subjective product design shape descriptor.

[0015] In some implementations, the digital component displays three or more different product design shapes for three or more different project types, feedback from the user regarding the project includes the selection of two or more different product design shapes that the user considers visually harmonious, and the method also includes using the feedback from the user in individual models.

[0016] In some implementations, the digital component requires the user to create a new product design, a feedback mechanism receives user input indicating the new product design, and the method further includes providing the user with a product featuring the new product design.

[0017] In some implementations, the digital component requires the user to modify an existing product design to produce a customized product design, the feedback mechanism receives user input to modify one or more aspects of the existing product design, and the method further includes providing the user with a product having the customized product design.

[0018] In some implementations, the digital component displays two distinct software attributes, feedback from the user regarding the project includes the selection of one of the two distinct software attributes preferred by the user, and the method further includes providing the user with a software package containing the selected one of the two distinct software attributes preferred by the user.

[0019] Other embodiments of this aspect include corresponding systems, apparatuses, and computer programs coded on computer storage devices that are configured to perform the actions of the method.

[0020] Specific embodiments of the subjects described in this document can be implemented to achieve one or more of the following advantages. In certain environments, there has previously been no way to automatically and systematically reduce bias in datasets that are underrepresented or identified as lacking data in a particular segment, and this drawback is addressed by the techniques, devices, and systems discussed herein.

[0021] In some implementations of this new system, underrepresented segments of the groups for which data is collected are identified, and tasks are generated to distribute to users within these segments. These tasks solicit responses from users regarding specific topics or domains, such that the responses supplement the existing dataset. For example, the system may determine that there are fewer than a threshold amount of responses from users within a specific age range regarding a preference between two products, and then generate tasks requiring users to choose between the two products to distribute to users within that age range. The system receives and processes these responses as a supplement to the existing dataset, thereby improving data quality. The system continuously monitors the dataset, providing a solution that automatically maintains data quality even when the dataset is updated.

[0022] This new system accesses sophisticated data processing infrastructure capable of cleaning, processing, and maintaining comprehensive, labeled, searchable datasets that can be used in many different situations to improve the results of models that previously lacked solutions, automatically improving and maintaining data quality as more data is collected. If a model uses an incomplete or underrepresented dataset, it may produce results that are insufficiently representative of actual group behavior. The new system automatically supplements the datasets from which the model derives, improving the robustness of the data and the accuracy of the model results. By automatically identifying group segments with insufficient or underrepresented data, the system reduces bias in the datasets used as input to various models. These more robust datasets, in turn, improve the reliability and accuracy of data-dependent model results. The improved datasets can be labeled and used by diverse parties, including content providers and product manufacturers who may not have the infrastructure to maintain the system's labeled and searchable datasets or cannot access them.

[0023] In addition to improving the dataset, the system can automatically explore the design space. The design space is a conceptual representation of design values ​​or attributes, and can be called a continuous shape. The design space can be associated with a specific product or service and can be multidimensional, representing possible design values ​​for parameters of interest. In some implementations, the design space can be enhanced to map design parameters to semantic values. For example, the system can use mappings of a product's semantic attributes and geometric features to create models that exist in a single, continuous shape space representing a system of possible attribute values. Designs can be based on one or more sensory features. For example, a design can be visual, auditory, tactile, odor-based, or taste-based. For example, the design of a car seat can include both visual and tactile features.

[0024] This new system automatically explores the design space by identifying segments of the design space with little or no data and generating tasks to distribute to users about those segments. By collecting data on these segments, the system allows for the consideration of previously unexplored designs. For example, the system can automatically generate designs with parameter values ​​that have not yet been presented to users for feedback. Based on the feedback received, the system can continuously update existing designs and generate new designs for which user feedback is sought before prototyping, manufacturing, and distribution. This feedback can serve as input to outputs such as models predicting user preferences. The system can determine the direction guiding product design based on the output of data analysis and behavioral models, allowing content providers, product designers, and manufacturers to focus on designs most likely to be welcomed by target consumer segments. In other words, the improved update process reduces the number of feedback cycles required to complete a product design, thereby reducing the computational resources required throughout the entire design cycle from initial data collection to design completion.

[0025] This approach allows for the rapid design and development of products that more accurately and reliably meet the needs and expectations of consumers across diverse demographics. Furthermore, the improved update process enhances the efficiency of the design system by increasing data quality and reducing the number of feedback cycles required to collect user data for input into behavioral models. The system provides product designers, developers, and manufacturers with a way to rapidly receive diverse feedback globally and determine user preferences based on demographics before the manufacturing and shipping stages—a process that typically requires significant resource expenditure. This approach can allow for personalized design and manufacturing for individual or user segmentation. For example, manufacturers can use this method to examine individual preferences to measure acceptance statistics across a larger market.

[0026] The technology described in this document enables the system to generate high-quality data relevant to a specific product or service using fewer resources and performing fewer operations. By automatically detecting and reducing biases in the dataset, the system allows models using the data to provide more accurate and reliable results. Furthermore, the system reduces the amount of resources required to complete the design of a new product or service by allowing targeted exploration of the design space and continuous improvement in data quality.

[0027] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0028] Figure 1 This is a block diagram of an example environment used for data quality improvement and design space exploration.

[0029] Figure 2A An example data flow of the data quality improvement process is shown.

[0030] Figure 2B and Figure 2C The model training process is described.

[0031] Figure 3 An example data flow of the data quality improvement process is shown.

[0032] Figure 4 An example data stream for designing a space exploration process is shown.

[0033] Figure 5A and Figure 5B The data flow describes a specific example of how the system uses a behavioral model to generate tasks for users.

[0034] Figure 6A and Figure 6BThe data flow is described as a specific example of how the system integrates user feedback into the design cycle.

[0035] Figure 7A and Figure 7B The data flow is described as a specific example of how the system implements user feedback to customize existing designs and products.

[0036] Figure 7C-7F It describes a specific example of how the system incorporates user feedback into the design cycle.

[0037] Figure 8 This is a flowchart of an example process for improving data quality.

[0038] Figure 9 This is a flowchart of an example process for automatically designing space exploration.

[0039] Figure 10 This is a block diagram of an example computing system.

[0040] The same reference numerals and names in different figures indicate the same elements. Detailed Implementation

[0041] This document describes methods, systems, and devices for improving data quality, reducing inherent bias, and enabling automated, intelligent exploration of the design space. The model is only as accurate and representative of the population as the input data provided to it. The proposed system improves the quality of data available to a variety of systems used for modeling and product development. User feedback and behavioral data can be collected in various ways.

[0042] In some implementations, the system generates tasks to be distributed to users. Each task can be a question, assignment, or other form of request for input from the user. User input can be tagged and used as part of a comprehensive database that can be used in different contexts. For example, a task can be presented to the user where they must select the single logo design that looks most appealing from multiple logo designs. The user's selection can then be tagged using the user's demographic information and stored as part of a tagged, searchable database. Based on analysis of the tagged dataset, the system can identify missing, insufficient, or underrepresented data from specific user demographics or regarding specific product segments, and automatically generate tasks or questions to gather more data and reduce inherent biases in underrepresented datasets. Supplementary tagged datasets can be provided as input to various models. For example, tagged datasets can be provided to behavioral models to predict whether a particular design will appeal to a specific user segment. Various models can be used to predict user responses and, for example, acceptance of a particular design.

[0043] The system also allows for automated and intelligent exploration of specific design spaces. For example, based on analysis of labeled datasets, the system can identify missing, insufficient, or underrepresented data related to specific areas of the design space, and automatically generate tasks or questions to collect more data. The system can explore the design space by generating product designs based on unexplored areas using artificial intelligence and machine learning models. These artificially generated designs can then be presented to the user along with tasks requesting feedback.

[0044] Figure 1 This is a block diagram of an example environment 100 for data quality improvement and design space exploration. Example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. Network 102 connects an electronic document server 104 (“Electronic Document Server”), user equipment 106, and a digital component distribution system 110 (also referred to as DCDS 110). Example environment 100 may include many different electronic document servers 104 and user equipment 106.

[0045] User equipment 106 is an electronic device capable of requesting and receiving resources (e.g., electronic documents) via network 102. Example user equipment 106 includes personal computers, mobile communication devices, and other devices that can send and receive data via network 102. User equipment 106 typically includes user applications such as web browsers to facilitate sending and receiving data via network 102, but local applications executed by user equipment 106 can also facilitate sending and receiving data via network 102.

[0046] One or more third parties 140 include content providers, product designers, product manufacturers, and other parties involved in the design, development, manufacture, marketing, or distribution of a product or service.

[0047] An electronic document is data that presents a set of content at user device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., "apps") such as those installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic document 105 ("electronic document") may be provided to user device 106 by electronic document server 104. For example, electronic document server 104 may include a server hosting a publisher's website. In this example, user device 106 may initiate a request for a given publisher's web page, and electronic document server 104 hosting the given publisher's web page may respond to the request by sending machine hypertext markup language (HTML) code that initiates the rendering of the given web page at user device 106.

[0048] Electronic documents can include a variety of content. For example, electronic document 105 may include static content (e.g., text or other specified content) that is within the electronic document itself and / or does not change over time. Electronic documents may also include dynamic content that may change over time or on a per-request basis. For example, the publisher of a given electronic document may maintain a data source for populating portions of the electronic document. In this example, a given electronic document may include tags or scripts that, when the given electronic document is processed (e.g., rendered or executed) by user device 106, cause user device 106 to request content from the data source. User device 106 integrates the content obtained from the data source into the presentation of the given electronic document to create a composite electronic document that includes the content obtained from the data source.

[0049] In some cases, a given electronic document may include a digital content tag or digital content script referencing DCDS110. In these cases, the digital content tag or digital content script is executed by user equipment 106 when the given electronic document is processed. Execution of the digital content tag or digital content script configures user equipment 106 to generate a request 108 for digital content, which is transmitted to DCDS110 via network 102. For example, the digital content tag or digital content script enables user equipment 106 to generate a packet data request including header and payload data. Request 108 may include data such as the name (or network location) of the server from which the digital content is requested, the name (or network location) of the requesting device (e.g., user equipment 106), and / or information that DCDS110 can use to select the digital content provided in response to the request. Request 108 is transmitted by user equipment 106 to the server of DCDS110 via network 102 (e.g., a telecommunications network).

[0050] Request 108 may include data specifying the characteristics of electronic documents and locations where digital content can be presented. For example, data specifying references to electronic documents (e.g., web pages) where digital content is presented (e.g., URLs), available locations within the electronic document for presenting digital content (e.g., digital content slots), the size of the available locations, the position of the available locations within the presentation of the electronic document, and / or the media types eligible for presentation in these locations may be provided to DCDS 110. Similarly, data specifying keywords may also be included in Request 108 (e.g., as payload data) and provided to DCDS 110 to facilitate the identification of digital content items eligible for presentation with electronic documents, the keywords being specified for selecting electronic documents (“document keywords”) or entities (e.g., people, places, or things) referenced by the electronic documents.

[0051] Request 108 may also include data related to other information, such as information provided by the user, geographic information indicating the state or region from which the request was submitted, or context indicating the environment in which the digital content will be displayed (e.g., the type of device on which the digital content will be displayed, such as a mobile device or tablet). Information provided by the user may include demographic data of the user of user device 106. For example, demographic information may include age, gender, geographic location, education level, marital status, household income, occupation, hobbies, social media data, and whether the user owns specific items, among other characteristics.

[0052] In situations where systems discussed here collect or may use personal information about users, users may be given the opportunity to control whether a program or function collects personal information (e.g., information about a user's social networks, social behaviors or activities, occupation, user preferences, or current location), or to control whether and / or how content that may be more relevant to the user is received from the content server. Furthermore, some data may be anonymized in one or more ways before storage or use, thereby removing personally identifiable information. For example, a user's identity may be anonymized, making it impossible to determine the user's personally identifiable information, or the user's geographic location may be generalized (such as city, zip code, or state level) when location information is obtained, making it impossible to determine the user's specific location. Therefore, users can control how information about them is collected and used by the content server.

[0053] Request 108 may also provide data specifying characteristics of user equipment 106, such as information identifying the model of user equipment 106, the configuration of user equipment 106, or the size (e.g., physical size or resolution) of the electronic display (e.g., a touchscreen or desktop monitor) displaying the electronic document. Request 108 may be transmitted, for example, via a packet network, and Request 108 itself may be formatted as packet data with a header and payload data. The header may specify the destination of the packet, while the payload data may include any information discussed above.

[0054] DCDS110 responds to receiving request 108 and / or uses information included in request 108 to select digital content to be presented with a given electronic document. In some embodiments, DCDS110 is implemented in a distributed computing system (or environment), which includes, for example, a server and a set of interconnected computing devices that identify and distribute digital content in response to request 108. This set of computing devices operates together to identify a set of digital content eligible for presentation in electronic documents within a corpus of millions or more available digital content. For example, millions or more available digital content may be indexed in a digital component database 112. Each digital content index entry may reference the corresponding digital content and / or include distribution parameters (e.g., selection criteria) that regulate the distribution of the corresponding digital content.

[0055] In some implementations, the digital components from the digital component database 112 may include content provided by a third party 140. For example, the digital component database 112 may receive photos of public intersections from a third party 140 that uses machine learning and / or artificial intelligence to navigate public streets. In another example, the digital component database 112 may receive specific questions that a third party 140, providing services to cyclists, wants users to respond to.

[0056] The identification of eligible digital content can be segmented into multiple tasks, and these tasks can then be distributed among computing devices within a set of multiple computing devices. For example, different computing devices in this set of multiple computing devices can each analyze different portions of the digital component database 112 to identify various digital content with distribution parameters that match the information included in request 108.

[0057] DCDS 110 aggregates results received from the group of multiple computing devices and uses information associated with the aggregation results to select one or more instances of digital content to be provided in response to request 108. DCDS 110 then generates and transmits response data 114 (e.g., digital data representing responses) via network 102, which enables user device 106 to integrate the selected set of digital content into a given electronic document, such that the selected set of digital content and the content of the electronic document are presented together on the display of user device 106. The digital content distributed by DCDS 110 and represented by response data 114 may include, for example, digital content requesting input from a user. This input can be analyzed, labeled, and stored as part of a comprehensive database such as a labeled database 130. The labeled database 130 stores labeled data that has been analyzed and categorized. The labeled database 130 can be searched and can store user-related data, including user demographics, user response data, and other user characteristics. For example, the labeled database 130 may store and associate anonymous user demographics with the user's responses to questions previously presented to the user. User input is transmitted from user device 106 to data quality processor 120 as response data 116.

[0058] Data quality processor 120 generates digital content that requests user input, receives and processes user input data, and generates and modifies design spaces and designs. Data quality processor 120 includes task processor 122, data processor 124, and model generator 126. Task processor 122 generates tasks to be distributed to users. Data processor 124 analyzes and tags the input received in response to the tasks. Model generator 126 generates and modifies design spaces and designs based on tagged data and input from third parties such as content providers, product designers, and manufacturers. For ease of explanation, Figure 1 The diagram illustrates task processor 122, data processor 124, and model generator 126 as separate components of data quality processor 120. Data quality processor 120 may be implemented as a single system on a computer-readable medium, which may be non-transitory. In some embodiments, one or more of task processor 122, data processor 124, and model generator 126 may be implemented as integrated components of a single system.

[0059] Task processor 122 creates digital content or tasks that request input from a user. Task processor 122 communicates with DCDS 110, electronic document server 104, and third party 140. Data quality processor 120 can collect data from tasks directly issued by task processor 122 or from tasks issued by third parties such as third party 140 that provide data quality processor 120 with access to its data sources. Tasks may include content requiring different levels of interaction, ranging from activities requiring a user to draw an image, to questions simply requiring a user to select an answer, to click activities requiring system permission to access user data. In some embodiments, a task may include a question requesting the user to answer input. For example, a task presented to a user may include the question “Do you prefer chocolate or vanilla ice cream?” and the user can enter their answer or choose from pre-selected answers. In some embodiments, a task may include activities requiring more user involvement. For example, a task presented to a user may require the user to select one or more portions of an image including a traffic intersection with bicycles, and the user can click or otherwise indicate the appropriate portion. In some implementations, the task may include a verification protocol challenge, such as CAPTCHA or reCAPTCHA.

[0060] In addition to generating tasks for users, task processor 122 can also modify tasks. For example, task processor 122 can modify a task that has previously been provided to one or more users and modify the task to collect different data, to formulate more targeted questions, or otherwise change the direction of the task. Task processor 122 and its output are described in more detail below.

[0061] Data processor 124 receives and processes data to identify missing, inaccurate, unrepresentative, or underrepresented data, and automatically determines data quality improvement solutions. Data processor 124 analyzes specific datasets and determines whether existing data meets quality thresholds based on design guidelines and other inputs. Data processor 124 can process response data received from user devices and existing response data. For example, data processor 124 can determine whether existing user response data stored in a labeled database 130 for a specific camping backpack design includes a representative number of responses from consumers aged 45 to 54 by determining the ratio of responses received from consumers in this age demographic to those in other age demographics and comparing the existing ratio to the expected or actual ratio in the camping backpack target market. Data processor 124 can also determine whether design values ​​within the design space have been explored or whether there is sufficient data on these values. For example, data processor 124 can determine whether a touchpad of a specific size on a laptop has received a sufficient number of user responses by comparing the existing number of user responses to a threshold number of user responses.

[0062] Data processor 124 can also receive and extract data collected during the digital content distribution process. For example, data analyzer 124 can receive request data 108 and response data 114 to determine the user group and characteristics represented by cookies indicated in the request data 108 and response data 114. Data analyzer 124 can store demographic and other characteristic data in a database, such as a tagged database 130. In some embodiments, data analyzer 124 can retrieve data from tagged database 130 that has been analyzed and tagged by other systems. Data analyzer 124 can, for example, retrieve data from tagged database 130 indicating demographic data of the user who provided request data 108 and received response data 114. Data processor 124 can, for example, segment the data based on user demographic information. Data processor 124 and its output are described in more detail below.

[0063] In addition to the above description, users may be provided with controls that allow them to choose whether and when a system, program, or feature described herein can collect user information (e.g., information about a user's social networks, social behaviors or activities, occupation, user preferences, or the user's current location), and whether to send content or communications to the user from a server. Furthermore, certain data may be processed in one or more ways before being stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed so that personally identifiable information about the user cannot be determined, or the user's geographic location may be generalized (such as city, zip code, or state level) when location information is obtained, making it impossible to determine the user's specific location. Therefore, users can control what information about themselves is collected, how that information is used, and what information is provided to them.

[0064] Model generator 126 generates, modifies, and maintains design spaces and models. Model generator 126 can generate design spaces and / or models based on data provided by data processor 124 or retrieved from a labeled database 130. For example, model generator 126 can generate behavioral models that predict user preferences for a particular design based on data provided by data processor 124. The behavioral models map subjective factors to continuous semantic shapes based on demographics, creating a design space that can be used to optimize the design within specific constraints.

[0065] The output of this model can range from a specific design to predicted user responses to that design. The model generator 126 and its output will be described in more detail below.

[0066] The techniques described below enable the system to continuously and automatically improve data quality and explore the design space.

[0067] Figure 2A It shows Figure 1 Example data flow 200 is an example of a data quality improvement process in an example environment. The operation of data flow 200 is performed by various components of system 100. For example, the operation of data flow 200 may be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tagged database 130.

[0068] The process begins in step A, where data processor 124 of data quality processor 120 analyzes and segments a dataset associated with a user group. In some implementations, the dataset is existing data retrieved from a labeled database 130. For example, the dataset may include a set of user inputs in response to a reCAPTCHA that asks a user to select all squares from a grid of photos showing a portion of a vehicle such as a bicycle. In another example, the dataset may include a set of user inputs in response to a question asking users how much they typically spend on dog food each month. User inputs may be associated with user characteristic data of the user providing the input. For example, user characteristic data may include demographic data and browsing history, as well as other data that system 100 can access and permits to use. In another example, the dataset may include a set of user inputs in response to a question asking users to rate their likelihood of purchasing a particular handbag design from a range of handbag designs mapped to subjective descriptors (e.g., “practical” or “fashionable”).

[0069] Data processor 124 segments the data based on various parameters, including user characteristic data. For example, data processor 124 may segment the dataset based on user age, user location, and / or user interests, as well as other user characteristic data that system 100 can access and permit system 100 to use. In another example, data processor 124 may segment the data based on characteristics of the data itself. For example, data processor 124 may segment the dataset based on the values ​​of specific subjective factors related to product design, such as the perceived "fashionability" of a handbag product design based on user feedback.

[0070] The process continues to step B, where data processor 124 identifies insufficient segmentation in the dataset based on one or more metrics. In some implementations, the metrics are provided by a third party 140, such as a product designer. For example, the metrics could be the number of responding users and the target demographic of those responding users. In some implementations, the metrics are determined automatically. For example, the metric could be a threshold difference in the proportion of responding users to their respective groups, where a group proportion difference greater than the threshold difference could be considered underrepresented from the actual group that requested the response. In one example, data processor 124 could identify bicycle detection segmentation in a vehicle detection set as having insufficient granularity.

[0071] The process continues to step C, where task processor 122 dynamically modifies the task to be presented to the user based on the identified segmentation. In some embodiments, task processor 122 dynamically modifies existing tasks that have been previously generated and / or presented to the user. In some embodiments, task processor 122 generates new tasks to be presented to the user. Task processor 122 can perform this modification in real time in response to the identified segmentation. For example, data quality processor 120 can continuously monitor the quality of the dataset and update its metrics based on new and updated information received.

[0072] In one example, task processor 122 automatically alters aspects of a previously distributed task by modifying the grid system to use smaller squares for better resolution. This task requires the user to select all squares containing a portion of a bicycle. In step B, data processor 124 automatically determines that a larger granularity is needed, and using this information, task processor 122 can divide the intersection photograph containing the bicycle into smaller squares.

[0073] Data quality processor 120 automatically modifies or generates new tasks based on analysis of existing datasets and performs additional operations that facilitate the generation of the tasks. For example, data quality processor 120 may determine that the modified task includes providing the user with photographic data of intersections where bicycles are visible. Data processor 122 may receive specific photographic data of intersections from a third party 140. Data processor 122 may also automatically locate sources of data to be provided as part of the task. For example, data processor 122 may retrieve photographic data from, for example, a labeled database 130 that is marked as having at least one bicycle in the field of view at public intersections. Data processor 122 may then perform data cleaning operations, including erasing personally identifiable information data, cleaning and adjusting data to make it usable, and other operations. For example, data processor 122 may adjust a live stream of photographs from street cameras trained at public intersections by operations such as filtering out images that do not include bicycles in the field of view, adjusting lighting, and creating a greater dynamic range in the images. Data processor 122 may perform complex data manipulation operations, including operations such as removing objects that obstruct another object and enhancing the focus of a specific object.

[0074] Task processor 122 can also determine one or more distribution parameters that must be met before a task can be distributed, based on the identified segmentation. Distribution parameters may include user characteristics that a user must possess in order for a task to be provided. For example, distribution parameters may include specific demographics of women aged 18 to 24 residing on the West Coast of the United States.

[0075] Task processor 122 can modify tasks to, for example, sample existing or new areas of space containing image, video, or audio objects within a tagged database 130. Task processor 122 can also test the removal or addition of brand information to, for example, assess user reactions to the brand or user preferences.

[0076] The process continues to step D, where task processor 122 transmits dynamically changed or generated tasks to DCDS 110 for distribution to users. For example, task processor 122 may transmit task data indicating the tasks to be presented to the user and the content to be presented to the user as part of the tasks. Task processor 122 may include distribution parameters that must be met in order to distribute tasks to specific users. For example, task processor 122 may include demographic data of the target user to whom the tasks can be presented.

[0077] The process continues to step E, where DCDS 110 receives a request 108 for content from user equipment 106. Request 108 is transmitted from user equipment 106 to DCDS 110 when the client device interacts with digital content. For example, if a user of user equipment 106 clicks a link to download a shopping app, the link may cause user equipment 106 to transmit request 108 to DCDS 110. Request 108 may include interaction tracking data from client equipment 106. For example, request 108 may include tracking data such as indications of the interaction, the digital content that user equipment 106 interacted with, and an identifier that uniquely identifies user equipment 106. In some implementations, request 108 includes indications of the digital content provider and the location of the target server hosting the requested resource.

[0078] The process continues to step F, where DCDS 110 transmits response data 114 to user equipment 106. As described above, response data 114, in addition to the requested electronic document, may indicate a task to be distributed to a user who meets specific distribution parameters. In response to DCDS 110 receiving request 108 and determining that the distribution parameters are met based on the received distribution parameters and the user data indicated in request 108, DCDS 110 transmits response data 114 to user equipment 106. For example, DCDS 110 may determine based on receiving request data 108 that the user of user equipment 106 is a 22-year-old female residing in Oregon, thus meeting the distribution parameters. DCDS 110 may then transmit the requested electronic document and dynamically changed task to user equipment 106 in the form of response data 114.

[0079] The process continues to step G, where DCDS 110 receives response data 116 from user device 106. In response to the user completing the task provided in response data 114, response data 116 is transmitted from user device 106 to DCDS 110. Response data 116 includes user information such as demographic data, device data, information about the user's response, and user input in response to the task provided in response data 114. For example, response data 116 may include the user's selection of a square containing a portion of a bicycle, the amount of time the user spent making the selection, her mouse movement pattern, and her anonymous demographic data, device data, and browsing history, all of which she has granted system 100 access. Response data may include semantic descriptors provided by the user regarding the task, product, or design. Semantic descriptors may include any descriptor that provides semantic information about an object such as a product or design. Semantic descriptors may be generated by humans or artificial intelligence and may take the form of words (e.g., keywords or key phrases), sentences, symbols, or other descriptors that convey semantic information. Furthermore, semantic descriptors can be assigned to objects based on other actions, such as interacting with presented information (e.g., photos or icons), interacting with rating elements (e.g., product rating tools), or submitting free-form textual feedback about the object. DCDS 110 can then provide this data to data processor 124 for analysis and labeling.

[0080] The process continues to step H, where data processor 124 analyzes response data 116 from user device 106. Data processor 124 can analyze response data 116 to categorize the data and tag it with user information, making the data searchable. For example, data processor 124 can tag the user's selected square with her demographic information, the amount of time she spent making the choice, and the accuracy of her choice compared to the true set of squares.

[0081] The process continues to step I, where data quality processor 120 provides the analyzed data to labeled database 130. Data processor 124 can provide labeled data for storage in labeled database 130, making the data searchable.

[0082] System 100 can continuously execute process 200, enabling data quality processor 120 to automatically and continuously monitor the quality of data included in a specific dataset. Therefore, system 100 maintains and improves the data quality in the comprehensive database 130, continuously improving the accuracy and completeness of the model output. Because system 100 continuously updates the labeled database 130, it provides a searchable database from which system 100 can retrieve content, such as image, audio, or video instances, based on user response information and user information such as task results.

[0083] The system reduces bias in the distribution of user feedback by selectively requesting additional feedback from user demographics that are not fully represented or not represented at all.

[0084] Figure 2B and Figure 2C The model training process is described. Server 250 maintains baseline model 252 and task repository 254. Baseline model 252 is the model used as the baseline behavioral model and can be updated. Task repository 254 maintains a set of tasks that can be distributed to users.

[0085] Devices A 260a, B 260b, ... and N 260n (collectively referred to as devices 260), each comprising models A, B, ... N262a, 262b, ... 262n (collectively referred to as models 262). Based on tasks and model updates provided by server 250, each locally maintained model 262 can be updated and improved.

[0086] Each of devices 260n receives input from users 270a, 270b, ... 270n (collectively referred to as users 270), displays information to the users 270a, 270b, ... 270n, and can be controlled by the users 270a, 270b, ... 270n. For example, device 260 can provide each user 270 with a task as described above. Tasks can be provided from, for example, a task repository 254. Each user 270 can provide semantic maps 272a, 272b, ... 272n (collectively referred to as semantic maps 272) to device 260 in response to a task. Based on the responses provided by users 270, device 260 can update model 262.

[0087] exist Figure 2C In this process, the model training module 256 can generate an updated baseline model 258 based on the information provided by the response from the user 270, and provide the updated baseline model 258 to the server 250 to replace or update the baseline model 252.

[0088] Figure 3The illustration depicts example design space 300. Design space 300 is a visual representation of a conceptual system of possible design values. Design space 300 can consist of, for example... Figure 1 The system generation of system 100 shown. For example, the model generator 126 of the data quality processor 120 can generate design space 300 based on user response data from a labeled database 130.

[0089] Design space 300 can be multi-dimensional. In this particular example, design space 300 includes two dimensions and is generated as a result of user-submitted data in response to a question asking users to rate various package designs. In other examples, design space 300 may include more than two dimensions and may be represented as a three-dimensional or more dimensional model. Dimensions include design features such as shape, color, texture, size, or relative distance to another object, as well as other features.

[0090] Design space 300 is used to visually represent all possible designs for a specific product or service. Products can include both non-durable and durable consumer goods. For example, products can include clothing, cosmetics, food, furniture, automobiles, and other items.

[0091] Design space 300 can be subject to various constraints, ranging from whether the design can be physically created to the ease of manufacturing of the design to constraints imposed by the designer. Design space 300 may be limited by data collected from sources such as the target audience of the product, including corporate users. For example, if the target consumer group for a particular mobile device would not carry it if it weighed more than three pounds, design space 300 could be limited in its weight dimension. In some implementations, design space 300 can be automatically limited by data and decisions made by system 100. For example, data processor 124 can determine, based on user response data from a tagged database 130, that none of the surveyed users are interested in laptops with screens smaller than 8 inches, and data processor 124 can provide this limitation on design space 300 to model generator 126.

[0092] By automatically generating and modifying the design space 300, this new system reduces the amount of time and resources required to reach the final design. The system can focus design exploration on areas where results are most likely to be achieved through metrics specified by interested third parties. For example, the data quality processor 120 can automatically focus design exploration for a bag design on areas most likely to be purchased by consumers aged 25 to 34—the target demographic of third-party bag designers and manufacturers 140. By automatically generating bag shapes and designs that fall within the design space 300 where users are most likely to be interested, the data quality processor 120 can focus user response data requests on bag designs that are most interesting to surveyed 25-34 year old users.

[0093] In this particular example, design space 300 maps semantic attributes to the geometric features of designs falling within that space. For example, design space 300 maps the semantic attributes “practical” and “fashionable” to the specific shapes and forms of bag designs including bag designs 302 and 304. These semantic attributes are subjective factors representing specific qualitative design goals and demographics. Based on user response data 306 and 308 from labeled database 130, data quality processor 120 has determined that users are most interested in bags that are a mix of “practical” and “fashionable.” In this particular example, data processor 124 of data quality processor 120 has analyzed user response data from labeled database 130 and determined that there is significant interest in bags that are more “practical” and at least somewhat “fashionable.” Using this determination, model generator 126 can limit design space 300 to focus on bag designs that exceed a certain threshold amount for “fashionable” and another threshold amount for “practical.” In some implementations, model generator 126 can automatically generate designs that meet these design criteria without further input from the designer. In some implementations, model generator 126 can generate package designs that meet the "fashionable" and "practical" thresholds that have not been generated before. For example, model generator 126 can generate package designs 312 and 314 without input from third-party package designer 140, and designs 312 and 314 can be new, previously unknown package designs.

[0094] Model generator 126 can generate new designs based on an initial set of input designs without additional information, by deforming specific geometric features of existing designs to create new designs, for example, using artificial intelligence and machine learning techniques. In some implementations, model generator 126 can generate entirely new designs without using existing designs as a starting point. Data quality processor 120 can be integrated with design programs, including 3D modeling programs and computer-aided design (CAD) programs, to generate new designs or modify existing designs.

[0095] Model generator 126 can use statistical and / or machine learning models that accept user-provided information as input. The machine learning model can use any of a variety of models, such as decision trees, generative adversarial network-based models, deep learning models, linear regression models, logistic regression models, neural networks, classifiers, support vector machines, inductive logic programming, model nesting (e.g., using techniques such as bagging, boosting, random forests, etc.), genetic algorithms, Bayesian networks, etc., and can be trained using a variety of methods, such as deep learning, association rules, inductive logic, clustering, maximum entropy classification, learned classification, etc. In some examples, the machine learning model can use supervised learning. In some examples, the machine learning model can use unsupervised learning.

[0096] In some implementations, a probability model may be used, which defines a probability ranking of an attribute of a given product design or shape, or a probability ranking of a product design or shape given one or more attributes.

[0097] Model generator 126 allows designers to explore new areas of the design space that have not been explored before, and reduces the amount of time and resource-intensive feedback cycles required for specific products by automatically generating new designs based on user response data, thereby increasing the likelihood that the design will be well-received by users. Designers can then select specific new designs for research and testing. In some implementations, probability ranking can be used to define a candidate set, which can then be tested in future tasks to refine user preferences. The data collection and design generation process is described in more detail below. Model generator 126 allows system 100 to characterize user behavioral requirements regarding product design based on market demographics.

[0098] The model generator 126 can use various types of models, including general models that can be used for all users and customized models that can be used for specific user subsets with shared feature sets, and can dynamically adjust the model based on received user information of a specific user 102 or based on detected activities. For example, the model generator 126 can use a base network for a user and then customize the model for each user.

[0099] In some implementations, model generator 126 may use machine learning to determine a target function for a specific user based on subjective feedback from the user. The target function can be simple and can be generically modified for product design. When system 100 collects user response data, system 100 anonymizes the data and provides it to a central database that stores and analyzes the collected data to improve the generic behavioral model and allow system 100 to provide more personalized strategies for each user 102.

[0100] For example, system 100 can utilize generic profiles of users with specific ages, locations, interests, etc. System 100 can provide profile support across users predicted to have similar interests. In some implementations, system 100 accepts input from user profile information, such as parameters like the user's age, location, and interests.

[0101] System 100 can utilize, for example, a somewhat personalized "shoe size" model. For instance, System 110 can use generic profiles for specific age groups, New Yorkers, motorcycle enthusiasts, etc. For example, if multiple users indicate similar preferences, System 100 can determine if a matching product exists. If a matching product exists, System 100 can return that product to the user. If no matching product exists, System 100 can modify an existing product design or generate a new design. In some implementations, System 100 can perform aggregated personalization or clustering by mapping specific users to existing user clusters or user segments, or by forming new user clusters or user segments. System 100 can also be used to identify product or purchasing trends within a specific user group. In some implementations, System 100 can perform aggregated configuration clustering by mapping users to existing customer segments or products, or by mapping user preferences to existing feature sets.

[0102] Furthermore, each model can be personalized. For example, each model can be created from a general model by changing model parameters based on characteristics of each user determined from collected data. For a specific user, each model can vary over long and short time periods. For example, system 100 can track a user's level of interest in a particular design element and adjust the behavioral model when it determines that a user has lost interest. In some implementations, each model can also be created from a model that has already been personalized using a general profile and further modified for each user. For example, a model can be created by changing model parameters based on characteristics of each user determined from collected data.

[0103] In some implementations, the model can be personalized without using a base model. For example, user response data can be input into model generator 126 and provided to product designers, manufacturers, or design programs to be mapped to product configurations without adjustment. In one example, model generator 126 allows a user to immediately purchase a specific item or set an alert when a specific item becomes available.

[0104] Figure 4 It shows Figure 1 Example data flow 400 is an example of the design space exploration process in the example environment. The operation of data flow 400 is performed by various components of system 100. For example, the operation of data flow 400 may be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tagged database 130.

[0105] The process begins in step A, where data processor 124 of data quality processor 120 analyzes and segments a dataset associated with a user group. In some implementations, this dataset is existing data retrieved from a labeled database 130. For example, the dataset may include a set of user responses to a question that asks users to rate two different bag designs on two semantic descriptors at a sliding ratio. User input may be associated with user characteristic data of the user providing the input. For example, user characteristic data may include demographic data and browsing history, as well as other data that system 100 can access and is authorized to use.

[0106] The process continues to step B, where data processor 124 identifies insufficient segmentation in the dataset based on one or more metrics. Details of this step can be found above regarding step B. Figure 2A It was found in the description.

[0107] The process continues to step C, where model generator 126 generates the design space. (As mentioned above...) Figure 3 As described, the design space is a visual representation of a possible design system. In some implementations, the model generator 126 simply updates an existing design space provided by a third party 140. For example, the model generator 126 may receive a design space from a third-party package designer 140 and update the design space based on a dataset.

[0108] Model generator 126 can also generate behavioral models that predict user acceptance of a particular design. For example, model generator 126 can utilize artificial intelligence and / or machine learning techniques to generate behavioral models that output assessments of the design space and the parts of the design space most likely to be welcomed by a particular user group, as well as other assessments of specific metrics for which there are insufficient data points or no representative proportion of users in a particular demographic segment.

[0109] The process continues to step D, where data processor 124 uses a behavioral model to determine one or more subdivisions of the design space as targets. For example, data processor 124 uses the output of the behavioral model generated in step C by model generator 126 to determine whether a design that is very “stylish” but also very “practical” has no threshold number of user responses to the design, or whether such a design has not yet been generated.

[0110] The process continues to step E, where task processor 122 dynamically changes the task to be presented to the user based on one or more determined subdivisions. (See above regarding...) Figure 2AAs described, task processor 122 can modify existing tasks that have been previously generated and / or presented to the user, or generate entirely new tasks to be presented to the user. For example, task processor 122 can generate a very "stylish" and also very "practical" new bag design to present to the user for feedback.

[0111] In some implementations, model generator 126 can display two or more product designs to allow users to visualize the differences or variations between two or more product instances generated by the model.

[0112] In some implementations, the model generator 126 may be integrated with a computer-aided design generation program, and the design of a product or service package may be improved, modified, or changed through the integration program.

[0113] The process continues to step F, where task processor 122 transmits dynamically changed or generated tasks to DCDS 110 for distribution to users. Details of this step can be found above regarding step D. Figure 2A It was found in the description.

[0114] The process continues to step G, where DCDS110 receives a request for content 108 from user equipment 106. Details of this step can be found above regarding step E. Figure 2A It was found in the description.

[0115] The process continues to step H, where DCDS110 transmits reply data 114 to user equipment 106. Details of this step can be found above regarding step F. Figure 2A It was found in the description.

[0116] The process continues to step I, where DCDS 110 receives response data 116 from user equipment 106. Details of this step can be found above regarding step G. Figure 2A It was found in the description.

[0117] The process continues to step J, where data processor 124 analyzes response data 116 from user equipment 106. Details of this step can be found above regarding step H. Figure 2A It was found in the description.

[0118] The process continues to step K, where model generator 126 updates the design space and / or behavioral model based on the analyzed response data from user device 106. Model generator 126 may narrow or expand the design space based on feedback from the user. For example, model generator 126 may discard a portion of the design space that has already been determined to have a threshold number or percentage of user responses and has positive responses below the threshold. Model generator 126 may update the behavioral model to reflect the updated dataset. For example, model generator 126 may input the analyzed response data as input to train a behavioral model predicting user acceptance of a particular package design.

[0119] The data quality processor 120 can also provide the analyzed data to the labeled database 130. Model and design space updates can be performed concurrently with the data quality processor 120 transferring the analyzed data to the labeled database 130. In some embodiments, these parts of step K can be performed asynchronously.

[0120] System 100 can continuously execute process 400, enabling data quality processor 120 to automatically and intelligently explore the design space. Therefore, system 100 allows for efficient design development, reducing the amount of time and resources required to generate and complete new designs for products or services.

[0121] As mentioned above Figure 1 and Figures 3-4 The described system automatically modifies tasks to provide data quality improvements. In some implementations, the data quality processor 120 may modify the task based on data indicating, for example, specific characteristics of the data segment itself, such as a lack of consistent response to one or more types of designs, different image locations, or images with specific characteristics, as well as other factors. In some implementations, the data quality processor 120 may modify the task based on data indicating, for example, specific characteristics in user segments, such as taking an unusually short amount of time to complete a task in a particular user segment or a lack of consistent response from the user segment, as well as other factors. System 100 automatically creates product designs based on consumer feedback and obtains further feedback on these designs, allowing for innovative designs that align with consumer preferences and needs without the need for designing and maintaining focus groups. Design improvements can be made more quickly with the help of more representative data. Furthermore, additional feedback can be used as input to, for example, networks used to train behavioral models to improve the model's classification of things that constitute positive or negative examples.

[0122] Figure 5A and Figure 5B The data flow of the system using a behavioral model to generate tasks for users is described.

[0123] Figure 5AThe data flow 500 is described, in which the system has user consent for personalized user settings. Figure 1 The tasks received in the example environment. The operation of data stream 500 is performed by various components of system 100. For example, the operation of data stream 400 may be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tagged database 130.

[0124] Process 500 begins with a person uploading one or more design elements and / or semantic descriptors, such as keywords or ways of expressing user preferences, such as user clicks or text input (502). The uploaded data can provide information that allows humans to express preferences, semantic descriptions, or ratings, which can be mapped to visual or auditory elements or features of a given design or more products or services being designed. For example, market researchers can provide system 100 with a backpack product design and a set of keywords associated with backpacks, such as “sports,” “functional,” “practical,” and “professional.” Design elements can be targeted at a product or service and can include designs for both physical and digital products. In another example, a task can present designs, descriptions, or features to determine the value or price of a product or service. These designs and / or keywords can be stored in a database, such as product designs and keywords. The person can also upload an experiment plan that determines which design classes or instances should be displayed with which semantic descriptions. The experiment plan can also include statistical measures or other guidelines on how, when, or where a given design type and semantic description should be displayed. In some implementations, the designs and / or keywords can be stored in a digital component database 112 and / or a tagged database 130.

[0125] Process 500 continues with the system selecting a format (504) for the content to be provided to the user. For example, the DCDS110 of system 100 can select a layout for the content to be provided to the user. For example, the layout may include the types of user interface elements available and the types of information provided. In one example, the DCDS110 can select a layout for the content that includes sliders and radio buttons for tasks to be provided to the user upon distribution. In another example, the format layout may include a reward to be provided to the user once the user has responded.

[0126] Process 500 continues with the system preprocessing data and verifying the correctness of the task to be provided to the user (506). For example, the task processor 122 and data processor 124 of the data quality processor 120 may preprocess data and verify the correctness of the task to be provided to the user based on the determined layout from (504).

[0127] Process 500 continues with the stored pre-processed data and verified tasks to be provided to the user (508). In process 500, system 100 obtains the user's consent to provide personalized content, and therefore can modify and / or personalize the pre-processed data and verified tasks based on user information.

[0128] Process 500 continues with the content interaction between the website or application's user and the service (510). For example, DCDS110 can select pre-processed data and verification tasks to be provided to specific users of the website or application, as described above. Figure 1 As described, it receives user input from the user's interaction with the content of the service.

[0129] Process 500 continues to store the user's response (512). For example, DCDS 110 can receive the user's response, which includes user information such as user demographics after the response data has been analyzed and processed by data processor 124 of data quality processor 120. The analyzed response data can be tagged by data processor 124 and stored in a tagged database 130.

[0130] Process 500 continues by building one or more behavioral models (514) based on user response data. Examples of behavioral models include mapping designs, design features, design variations, or graph-based design representations to semantic descriptions or ratings using linear or nonlinear function approximations (e.g., convolutional neural networks that enable deep learning). Reversible models that allow mapping from input to output and vice versa can be used, or multiple models can be used to map input to output and vice versa. Behavioral models may also include user demographic information stored along with the responses of a given user. One example is creating a model using task data that allows people to view clothing design feedback from women aged 20 to 30 living in London. For example, a behavioral model may take user demographics as input, or be essentially a probabilistic model, allowing a computer to adjust the model response for a given set of demographic parameters. For example, the model generator 126 of the data quality processor 120 may generate behavioral models based on user response data, as described above. Figure 1-4 As described. Model-based or model-driven analytics may also include information or input from other sources, such as online surveys, pricing and conversion data, sales attribution models, topical clinics, and focus group feedback. Model-based or model-driven analytics may also utilize pricing information obtained from task feedback.

[0131] Process 500 continues with the analysis and identification (516) of new ideas and / or concepts based on one or more behavioral models. For example, the model generator 126 of the data quality processor 120 can analyze and identify new design ideas and / or new design concepts based on the output of the behavioral models.

[0132] Process 500 continues by exploring and / or optimizing the design to generate new designs and / or keywords or semantic descriptors or preferences (518). For example, the model generator 126 of the data quality processor 120 can use the design space to explore and / or optimize the design to generate new designs or modify existing designs. The model generator 126 can also generate new keywords associated with the design. For example, the model generator 126 can generate new keywords that are trendy or popular in current marketing materials and that have been shown to be popular among users based on the generated behavioral model. Models can be used to generate designs from semantic descriptions or ratings and can be used to generate semantic descriptions or ratings from a given design. These models can be used to explore or optimize the model space to identify new designs or semantic descriptions to be included in future tasks. Figure 3 See the example below, which illustrates the model space where designs are positioned relative to two axes associated with keywords (fashion and practicality). In this example, optimization with a cost function weighted 60% for practicality and 40% for fashion might select design 312. One example includes a behavioral model that incorporates pricing feedback, ensuring that optimization can take into account design trade-offs, costs, and pricing tasks. These new designs and / or keywords are provided to a database of product designs and keywords referenced in (502). In some implementations, designs and / or keywords may be stored in a digital component database 112 and / or a tagged database 130.

[0133] Figure 5B Data flow 550 is described, in which the system does not have personalized user information. Figure 1 User consent is received for tasks in the example environment. The operation of data stream 550 is performed by various components of system 100. For example, the operation of data stream 550 may be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tagged database 130.

[0134] Process 550 follows directly after Process 500. Process 550 begins with the client uploading one or more design elements and / or keywords (552). Details of this step can be found above regarding (502). Figure 5A It was found in the description.

[0135] Process 550 continues with the content selection format (554) to be provided to the user by the system. Details of this step can be found above regarding (504). Figure 5A It was found in the description.

[0136] Process 550 preprocesses the data and verifies the correctness of the task to be provided to the user (556). Details of this step can be found above regarding (506). Figure 5A It was found in the description.

[0137] Process 550 continues storing the pre-processed data and the verification task to be provided to the user (558) and continues. In process 550, system 100 has not provided user consent for personalized content, therefore the pre-processed data or verification task will not be modified or personalized based on user information.

[0138] Process 550 continues with the user's interaction with the service content on the website or application (560). Details of this step can be found above regarding (510). Figure 5A It was found in the description.

[0139] Process 550 continues with storing the user's response (562). For example, DCDS 110 may receive a user's response, which may include user information such as user demographics after the response data has been analyzed and processed by data processor 124 of data quality processor 120. DCDS 110 may remove user information if it is included. In some embodiments, user information is not provided because system 100 does not have the user's consent to access the user's information. The analyzed response data may be tagged by data processor 124 and stored in a tagged database 130.

[0140] Process 550 continues by building one or more behavioral models (564) based on user response data. Details of this step can be found above regarding (514). Figure 5A It was found in the description.

[0141] Process 550 continues with the analysis and identification (566) of new ideas and / or concepts based on one or more behavioral models. Details of this step can be found above regarding (516). Figure 5A It was found in the description.

[0142] Process 550 continues with exploring and / or optimizing the design to generate new design and / or semantic descriptors (568). Details of this step can be found above regarding (518). Figure 5A It was found in the description.

[0143] Figure 6A and Figure 6B The data flow of integrating user feedback into the design cycle is described. Figure 6A and Figure 6B Based on Figure 5A and Figure 5B The data stream is depicted in the diagram. In some embodiments, system 100 has user consent for personalized content. In some embodiments, system 100 does not have user consent for personalized content.

[0144] Figure 6A The data flow 600 is described, in which the system integrates user feedback into the design cycle and undergoes [further processing / processing]. Figure 1 The example environment is designed with the input of the designer. The operation of data flow 600 is performed by various components of system 100. For example, the operation of data flow 600 may be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tagged database 130.

[0145] Process 600 follows either process 500 or 550. Process 600 begins with the user selecting the task format and uploading design primitives and semantic descriptors (602). Details of this step can be found above regarding (502). Figure 5A Or (552) Figure 5B The description can be found there. For example, users can upload design elements or shapes along with related keywords to the database. This database stores the initial product design shapes and keywords, as well as any updates based on user responses to the viewing task. For example, the database can store updates based on step 618.

[0146] Process 600 continues with the content selection format (604) to be provided to the user by the system. Details of this step can be found above regarding (504). Figure 5A Or about (554) Figure 5B The description can be found therein. The format of the primitives and semantic descriptors stored can be chosen by a person or configured automatically by the system.

[0147] Process 600 continues with the system preprocessing data and verifying the correctness of the task to be provided to the user (606). Details of this step can be found above regarding (506). Figure 5A Or about (556) Figure 5B The description can be found within the context of the task. For example, the data can be preprocessed to identify a set of keywords suitable for a specific design shape. Furthermore, the data can be validated for appropriateness or correctness for a given format or desired user audience. In some implementations, the task may be relevant to specific country, geographic, or demographic characteristics. For example, the interpretation of a French description might be irrelevant to the task for a large portion of non-French-speaking users in the US and Europe. These tasks can be stored and provided along with content distributed via the Internet or through an application.

[0148] Process 600 continues with the stored preprocessed data and verified task to be provided to the user (608). Details of this step can be found above regarding (508). Figure 5A Or about (558) Figure 5B It was found in the description.

[0149] Process 600 continues with the user's interaction with the service content on the website or application (610). Details of this step can be found above regarding (510). Figure 5A Or about (560) Figure 5BThe description can be found there. For example, a user interacts with the content and makes a request for a task. Then, the user can interact with the task and provide a response. For example, a user can select the most suitable keywords or tags for a photo or image of a product design.

[0150] Process 600 continues with storing the user's response (612). Details of this step can be found above regarding (512). Figure 5A Or about (562) Figure 5B The description can be found there. In some implementations, only the response is logged. In some implementations, both the response and user demographics may be stored.

[0151] Process 600 continues by building one or more behavioral models (614) based on user response data. Details of this step can be found above regarding (514). Figure 5A Or about (564) Figure 5B The description can be found within the context of the design. For example, a model can map keywords and semantic descriptors to specific design shapes or geometries. In another example, a model can map product features to other sets of preferred features or products to other sets of preferred products. These models can also map design features or decisions to customer preferences and values.

[0152] Process 600 continues with the analysis and identification (616) of new ideas and / or concepts based on one or more behavioral models. Details of this step can be found above regarding (516). Figure 5A Or about (566) Figure 5B The description can be found there. For example, analyzing and identifying models to generate data and results that can be used to help determine new tasks.

[0153] Process 600 continues with exploring and / or optimizing the design to generate new designs and / or keywords (618). Details of this step can be found above regarding (518). Figure 5A Or about (568) Figure 5B The description can be found there. For example, the system can respond to analysis by using data to explore and optimize or modify task content, format, or messages. Any learning, modifications, and updates related to model, analysis, and design decisions and optimizations are stored in the database.

[0154] Process 600 includes loading a behavioral model into a CAD-based design tool and making product design decisions based on the behavioral model (620). In some implementations, human designers can provide input and / or product design decisions to the CAD-based design tool. For example, human designers may implicitly have a desired semantic description of the outcome in mind and can shape the output of the CAD-based design tool and automatically generate product design decisions made by model generator 126. The CAD-based design tool can also receive constraints and specifications provided by, for example, human designers, product manufacturers, or customers. In some implementations, models that capture user feedback or behavior can be used directly or indirectly for computer-aided design decisions and manufacturing production. These models can be used to analyze or modify the design.

[0155] Process 600 continues to generate a modified design shape (622). For example, model generator 126 can use a behavioral model and the product's design space to generate a modified design shape. In some implementations, the model can map keyword descriptions to actual 3D product designs, and designers can use keyword descriptions and user feedback to generate new or modified designs. The design can be represented as a defined shape, a data structure that represents a shape or deformation as a given shape or shape diagram. In one example, the designer uses the model to create or modify the design based on other products that may be sold together or bundled with it. This loop can iterate until product requirements or release criteria are met. If they are met, the design is provided as a specification, manufacturing guide, recipe, or method of guiding or directing the manufacturing process. The resulting product can be sent to the original user who provided feedback or provided to a warehouse or store for distribution to customers. In some implementations, this method can be provided manually, where information is provided to the person making the decision. In some implementations, the method can be used to automatically modify the design based on a cost function of constraints, specifications, or guidance toward a better state. In some implementations, this method can use a mix of steps that are performed by humans and those that are automated.

[0156] Process 600 continues to determine whether the modified design shape meets the release criteria (624). For example, model generator 126 may determine whether the design shape is suitable for release or manufacturing based on release criteria from, for example, product designers, manufacturers, transporters, etc.

[0157] Process 600 may optionally include manufacturing and shipping products with modified design shapes (626). For example, the design process may be integrated with the product manufacturing and distribution workflow. In some implementations, products may be distributed to warehouses, stores, and directly to users with similar preferences to the user who provided the initial response, such as other users within the same group, cluster, location, or user segment, and other shared groups.

[0158] Figure 6B The data flow 650 is described, in which the system integrates user feedback into the design cycle to... Figure 1 New designs are automatically generated in the example environment. The operation of data flow 650 is performed by various components of system 100. For example, the operation of data flow 650 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tagged database 130.

[0159] A key feature of process 650 is that generative design can be used to enhance the vision of human designers, and in some implementations, it can completely replace human designers based on the desired results.

[0160] Process 650 follows directly after processes 500, 550, or 600. Process 650 begins with the client uploading one or more design elements and / or keywords (652). Details of this step can be found above regarding (502). Figure 5A Regarding (552) Figure 5B Or about (602) Figure 6A It was found in the description.

[0161] Process 650 continues with the content selection format (654) to be provided to the user by the system. Details of this step can be found above regarding (504). Figure 5A Regarding (554) Figure 5B Or about (604) Figure 6A It was found in the description.

[0162] Process 650 continues with the system preprocessing data and verifying the correctness of the task to be provided to the user (656). Details of this step can be found above regarding (506). Figure 5A Regarding (556) Figure 5B Or about (606) Figure 6A It was found in the description.

[0163] Process 650 continues with the stored preprocessed data and verified task to be provided to the user (658). Details of this step can be found above regarding (508). Figure 5A Regarding (558) Figure 5B Or about (608) Figure 6A It was found in the description.

[0164] Process 650 continues with the interaction (660) between the website or application's user and the content being served. Details of this step can be found above regarding (510). Figure 5A Regarding (560) Figure 5B Or about (610) Figure 6A It was found in the description.

[0165] Process 650 continues with storing the user's response (662). Details of this step can be found above regarding (512). Figure 5A Regarding (562) Figure 5B Or about (612) Figure 6A It was found in the description.

[0166] Process 650 continues by building one or more behavioral models (664) based on user response data. Details of this step can be found above regarding (514). Figure 5A Regarding (564) Figure 5B Or about (614) Figure 6A It was found in the description.

[0167] Process 650 continues with the analysis and identification (666) of new ideas and / or concepts based on one or more behavioral models. Details of this step can be found above regarding (516). Figure 5A Regarding (566) Figure 5B Or about (616) Figure 6A It was found in the description.

[0168] Process 650 continues with exploring and / or optimizing the design to generate new design and / or semantic descriptors (668). Details of this step can be found above regarding (518). Figure 5A Regarding (568) Figure 5B Or about (618) Figure 6A It was found in the description.

[0169] Process 650 includes loading the behavioral model into a CAD-based design tool and making product design decisions based on the behavioral model (670). In some implementations, generative design processes can be used to automatically generate or modify designs based on desired criteria or cost functions. Such generative design processes can provide input to CAD-based design tools and / or provide product design decisions. For example, a generative design system can determine or receive an appropriate cost function based on desired results. CAD-based design tools can also receive constraints and specifications provided by, for example, human designers, product manufacturers, or customers. For example, artificial intelligence methods can be used in conjunction with iterative generative design to search the design space and produce designs that conform to specifications, as well as results from the behavioral model.

[0170] Process 650 continues to produce the modified design shape (672). Details of this step can be found above regarding (622). Figure 6A It was found in the description.

[0171] Process 650 proceeds to determine whether the modified design shape meets the release criteria (674). Details of this step can be found above regarding (624). Figure 6A It was found in the description.

[0172] Process 650 may optionally include manufacturing and shipping the product with the modified design shape (676). Details of this step can be found above regarding (626). Figure 6A It was found in the description.

[0173] In one example, the task might include showing a user two different products, such as shoes, and asking the user which shoe matches the shown outfit better. For example, the outfit could be a suit. The outfit could include multiple different clothing items, and the user would be asked to choose a set.

[0174] In one example, a task could include a way for users to configure or modify a design and request feedback on how to create a new product or improve an existing one. The improved product can then be offered to users in new tasks, who can iteratively refine the product design.

[0175] In one example, the task could include showing a user two different product designs, such as a car design, and providing the user with feedback mechanisms such as sliders or buttons to provide a rating of how much the user approves of a given attribute, such as a subjective product design descriptor (or adjective) like “compact” to describe each design.

[0176] Computer-aided design processes can be automated to generate and modify designs according to standards and specifications. Cost functions can be used to weigh design criteria or factors within constraints and specifications. Behavioral models can be used in conjunction with cost functions to maximize user preferences within constraints, specifications, or manufacturing rules. For example, generative design methods can be used to iterate across a design space guided by behavioral models and cost functions or decision criteria. The use of generative design involves using AI and reinforcement learning to iterate across numerous design decisions—such as the body style of a car or the color combination of running shoes—that are permissible or within specifications or constraints but may represent different customer preferences. Generative algorithmic methods include one or more of evolutionary algorithms, variational autoencoders, and generative adversarial networks. These methods can leverage cloud computing to iterate a large number of design iterations to optimize for a single customer, customer segment, or multiple customer segments. Examples of algorithms that can be used include evolutionary algorithms, including genetic algorithms that evolve a given design or create a hybrid of multiple designs. In some implementations, the system can use a generative adversarial network that takes multiple existing designs as input to generate entirely new designs. These methods can iterate to generate new designs, and the task is presented to the user to provide feedback on said new design. The entire process can be automated through iteration and optimization to generate new designs. In some implementations, the system can be integrated into automated manufacturing processes using robotics or 3D printing. In other implementations, the system can be used to personalize products for users or user segments based on preferences provided by the user or user segment.

[0177] Variational autoencoders and associated generative adversarial networks can be used with task-generated behavioral models to evolve new designs, which are then fed into the system to obtain additional user feedback. The system iterates and scores until a stopping criterion defined by the cost function is met.

[0178] Variational autoencoders can be used with design libraries represented as images, shapes, shape-related data structures, shape-related graphs, or polygon data. Variational autoencoders can be used to create a set of latent factors that effectively represent a reduced set of features describing a given design. Once the variational autoencoder is trained, the design is encoded into its latent factors and then decoded into its reconstructed design. In one example, an existing design representation library can be used to build an initial mapping to accelerate convergence. Behavioral models developed using task-based feedback can be used to map a given reconstructed design representation to a user representation or classification. In some implementations, user classification can be a rating of good (preference) versus bad (non-preference). User / human-based representations can be used with a cost function to create an optimizer that can be used to create or improve a mathematical representation or score of the design. The optimizer's output is the design type associated with the latent factor description. Optimization can even use mutation or crossover to create new latent factors. New latent factors can be passed through a decoder / generator to produce new design representations that can be inserted into new tasks. Designs, latent factors, and keywords all form a design space and a distance metric that can be used to correlate or cluster one design with another when creating a new task presented to a user. The system can automatically iterate around the design space until the cost-based scoring converges, where the resulting latent factors meet the stopping criteria or require very little modification. At this point, the design is considered complete and ready for production.

[0179] In another implementation, creating or optimizing a design may include creating a mathematical representation of a design space, where each car or clothing design is represented by design features defined by each dimension of the space. The N-dimensional feature space is represented in an N-space. In many cases, the space can be transformed into a low-dimensional space. In other cases, distance metrics can be used to cluster or segment design instances into classes. The distance from each design instance or class to another can be calculated and stored. The behavioral model represents a mapping between design instances and a set of keywords or semantic descriptions for the task evaluator / user. This mapping can be explored to find designs similar to the most promising designs rated so far (optimization) or designs dissimilar to those (exploration) to discover and explore new parts of the design space. The system can be used in conjunction with distance-based clustering methods—such as K-nearest neighbors or collaborative filtering—to define design instances for future tasks presented to the user.

[0180] Figure 7A and Figure 7B The data flow for implementing user feedback to customize existing designs and products is described.

[0181] Figure 7A Data flow 700 is described, in which the system implements user feedback for modification. Figure 1The example environment includes existing designs and products. The operation of data flow 700 is performed by various components of system 100. For example, the operation of data flow 700 may be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tagged database 130.

[0182] Dataflow 700 can be integrated with various manufacturing processes, including 3D printing or automated manufacturing.

[0183] Process 700 follows immediately after processes 500, 550, 600, or 650. Process 700 begins with the client uploading one or more design elements and / or keywords (702). Details of this step can be found above regarding (502). Figure 5A Regarding (552) Figure 5B Or about (602) Figure 6A Regarding (652) Figure 6B It was found in the description.

[0184] Process 700 continues with the content selection format (704) to be provided to the user by the system. Details of this step can be found above regarding (504). Figure 5A Regarding (554) Figure 5B Regarding (604) Figure 6A Or about (654) Figure 6B It was found in the description.

[0185] Process 700 continues with the system preprocessing data and verifying the correctness of the task to be provided to the user (706). Details of this step can be found above regarding (506). Figure 5A Regarding (556) Figure 5B Regarding (606) Figure 6A Or about (656) Figure 6B It was found in the description.

[0186] Process 700 continues with the stored preprocessed data and verified task to be provided to the user (708). Details of this step can be found above regarding (508). Figure 5A Regarding (558) Figure 5B Regarding (608) Figure 6A Or about (658) Figure 6B It was found in the description.

[0187] Process 700 continues with the user's interaction with the service content on the website or application (710). Details of this step can be found above regarding (510). Figure 5A Regarding (560) Figure 5B Regarding (610) Figure 6A Or about (660) Figure 6BIt was found in the description.

[0188] Process 700 continues with storing the user's response (712). Details of this step can be found above regarding (512). Figure 5A Regarding (562) Figure 5B Regarding (612) Figure 6A Or about (662) Figure 6B It was found in the description.

[0189] Process 700 continues by building one or more behavioral models (714) based on user response data. Details of this step can be found above regarding (514). Figure 5A Regarding (564) Figure 5B Regarding (614) Figure 6A Or about (664) Figure 6B It was found in the description.

[0190] Process 700 continues with the analysis and identification (716) of new ideas and / or concepts based on one or more behavioral models. Details of this step can be found above regarding (516). Figure 5A Regarding (566) Figure 5B Regarding (616) Figure 6A Or about (666) Figure 6B It was found in the description.

[0191] Process 700 continues with exploring and / or optimizing the design to generate new design and / or semantic descriptors (718). Details of this step can be found above regarding (518). Figure 5A Regarding (568) Figure 5B Regarding (618) Figure 6A Or about (668) Figure 6B It was found in the description.

[0192] Process 700 includes using a behavioral model to identify existing products or product suites that have features that most closely follow the design identified by the behavioral model (720). In some embodiments, the model generator 126 may access a database or other searchable structure that stores existing product designs and features with or without pre-existing keyword descriptions.

[0193] Process 700 continues to produce the modified design shape (722). Details of this step can be found above regarding (622). Figure 6A Or about (672) Figure 6B It was found in the description.

[0194] Process 700 proceeds to determine whether the modified design shape meets the release criteria (724). Details of this step can be found above regarding (624). Figure 6A Or about (674) Figure 6B It was found in the description.

[0195] Process 700 may optionally include manufacturing and shipping the product with the modified design shape (726). Details of this step can be found above regarding (626). Figure 6A Or about (676) Figure 6B It was found in the description.

[0196] In one example, the task involves displaying a suite of product designs. For instance, the suite could include different product designs for different item types, such as room furniture (e.g., chairs, tables, beds, accessories, artwork, etc.), and ask the user to select which of the different product designs they find visually appealing along with the different item types. In some implementations, user feedback can be used in subsequent tasks to test existing and develop new marketing strategies, such as cross-selling and upselling strategies.

[0197] In another example, the task includes how a user designs or configures a product such as shoes or a car, and how a user receives the product or how a manufacturer produces and ships the product to the user. For example, a user could provide input through a feedback mechanism to indicate a new design or a design configured using a predetermined set of characteristic values.

[0198] In another example, the task involves users modifying existing product designs and having manufacturers provide users with ways to customize products (e.g., using a 3D printer). For instance, users could provide input through a feedback mechanism to indicate modifications to the existing design.

[0199] Figure 7B Data flow 750 is described, in which the system implements user feedback to... Figure 1 Custom software design and products are provided in the example environment. The operation of data flow 750 is performed by various components of system 100. For example, the operation of data flow 750 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tagged database 130.

[0200] Dataflow 750 uses semantic behavior models to configure software products to be customized for, for example, specific users, user segments, or groups. In one example, Dataflow 750 is used to create mini-games with characters, weapons, environments, and / or situations, and other features, based on user information selection or customization.

[0201] Process 750 follows processes 500, 550, 600, 650, or 700. Process 750 begins with the customer uploading one or more design elements and / or semantic descriptors (752). Details of this step can be found above regarding (502). Figure 5A Regarding (552) Figure 5B Regarding (602) Figure 6A Regarding (652) Figure 6B Or about (702) Figure 7A It was found in the description.

[0202] Process 750 continues with the system selecting the format of the content to be provided to the user (754). Details of this step can be found above regarding (504). Figure 5A Regarding (554) Figure 5B Regarding (604) Figure 6A Regarding (654) Figure 6B Or about (704) Figure 7A It was found in the description.

[0203] Process 750 continues with the system preprocessing the data and verifying the correctness of the task to be provided to the user (756). Details of this step can be found above regarding (506). Figure 5A Regarding (556) Figure 5B Regarding (606) Figure 6A Regarding (656) Figure 6B Or about (706) Figure 7A It was found in the description.

[0204] Process 750 continues with the stored preprocessed data and verified task to be provided to the user (758). Details of this step can be found above regarding (508). Figure 5A Regarding (558) Figure 5B Regarding (608) Figure 6A Regarding (658) Figure 6B Or about (708) Figure 7A It was found in the description.

[0205] Process 750 continues with the user's interaction with the service content on the website or application (760). Details of this step can be found above regarding (510). Figure 5A Regarding (560) Figure 5B Regarding (610) Figure 6A Regarding (660) Figure 6B Or about (710) Figure 7A It was found in the description.

[0206] Process 750 continues with storing the user's response (762). Details of this step can be found above regarding (512). Figure 5A Regarding (562) Figure 5B Regarding (612) Figure 6A Regarding (662) Figure 6BOr about (712) Figure 7A It was found in the description.

[0207] Process 750 continues by building one or more behavioral models (764) based on user response data. Details of this step can be found above regarding (514). Figure 5A Regarding (564) Figure 5B Regarding (614) Figure 6A Regarding (664) Figure 6B Or about (714) Figure 7A It was found in the description.

[0208] Process 750 continues with the analysis and identification (766) of new ideas and / or concepts based on one or more behavioral models. Details of this step can be found above regarding (516). Figure 5A Regarding (566) Figure 5B Regarding (616) Figure 6A Regarding (666) Figure 6B Or about (716) Figure 7A It was found in the description.

[0209] Process 750 continues with exploring and / or optimizing the design to generate new design and / or semantic descriptors (768). Details of this step can be found above regarding (518). Figure 5A Regarding (568) Figure 5B Regarding (618) Figure 6A Regarding (668) Figure 6B Or about (718) Figure 7A It was found in the description.

[0210] Process 750 includes using behavioral models to configure software or digital products for personalization or customization based on user preferences (770). In some implementations, a configurator is used to personalize the software product. In some implementations, the software product may be a mini-game. In some implementations, the software product may be embedded in a web page or application.

[0211] For example, if user information indicates that a particular user likes the Star Wars series and the software product is a mini-game, then the data quality processor 120, and in particular the model generator 126, can modify the mini-game with the permission of the user and Lucasfilm to include characters, audio, items, etc. from the Star Wars series.

[0212] In another example, game developers take an existing or new game that may have many levels, characters, and weapons, and allow users to select parts that can be configured with user feedback through missions, allowing a small portion to be played online or downloadably.

[0213] Step 750 proceeds to determine if the modified game design meets release criteria (772). Details of this step can be found above regarding (624). Figure 6A Regarding (674) Figure 6B Or about (724) Figure 7A It was found in the description.

[0214] Process 750 may optionally include distributing or providing a download option with the modified game design to users interacting with the system, providing feedback (774), and actually delivering the modified game design to other users (775). Details of this step can be found above regarding (626). Figure 6A Regarding (676) Figure 6B Or about (726) Figure 7A It was found in the description.

[0215] In one example, the task involves showing a user two different software game characters or game application-related attributes such as window size and asking the user which one they are more interested in. In some examples, the method may also include providing software pre-configured with the selected character or attributes for the user to download. In another example, the method may include artifacts for users to play games online using software pre-configured with the selected character or attributes. In yet another example, behavioral models are created using task feedback from one or more users to design new software such as an app or game, which may have various forms utilizing characters, settings, and backgrounds tailored to the demographics of one or more users. In yet another example, multiple users can use tasks to collaboratively design game features that are then integrated into a multiplayer game environment accessible to the users.

[0216] Process 750 may optionally include distributing or providing download options with the modified game design to users other than the user who is interacting with the system and providing feedback (776), and actually delivering the modified game design to other users (777). Implementation details are the same as (774).

[0217] Figure 7C Here is a specific example where the system implements user feedback as shown in the example regarding... Figure 7BIn the described design cycle, the system can receive products such as software titles (780). For example, the system can receive games or applications. The system can identify the product's attributes and assets. For example, the system can determine the game's attributes, including characters, weapons, scenes, environments, keywords, etc. The system can identify the game's assets, including thumbnails, demo versions, multiple configurations, usage / game videos, reviews, descriptions, keywords, hyper-casual versions, etc. The system can present the game's attributes to the user (782). For example, it can present character A, character B, weapon A, and weapon B to the user. The user can make selections and choose to play the game. The system can present the game's assets to the user (783). For example, it can present thumbnails to the user. Figure 1 The screen includes a thumbnail and two sliders that prompt the user for input. The first slider asks, "Which game is this label: 'Role Playing'?" The user can drag the slider to the side representing their answer. The second slider asks, "Which game is this label: 'Puzzle'?"

[0218] Figure 7D Here is a specific example where the system implements user feedback as shown in the example regarding... Figure 7B In the described design cycle, the user can interact with the task to select settings or configuration preferences (784). Task-related UI elements can be presented to the user (785). The game or application can be configured using the processes outlined in (786)-(786). For example, the process may include settings, configurations, or game state descriptions such as character / skin, inventory / load, settings / environment, and task (786). These settings and configurations may be encrypted to control the use of subgames and restrict their permissions. The engine can then use the settings, configurations, or game state descriptions to take input and generate a usable and workable environment (787). The game or application can then be a configured / personalized experience (788). The configured game or application can be provided to the user via task-related UI elements (789). The user can then receive or download the configured game or application (790). The user can be the same user who interacted with the task in (784) or a different user.

[0219] Figure 7E Here is a specific example where the system implements user feedback as shown in the example regarding... Figure 7BIn the described design cycle, the user can interact with the task to select settings or configuration preferences (791). Task-related UI elements can be presented to the user (792). The game or application can be configured using the processes outlined in (793)-(795). For example, the process may include settings, configurations, or game state descriptions such as character / skin, inventory / loading, settings / environment, and task (793). These settings and configurations may be encrypted to control the use of subgames and restrict their permissions. The engine can then map the user's settings, configurations, or game state preferences to pre-configured or pre-compiled games / applications from the library (794). The game or application can then be a configured / personalized experience (795). The configured game or application can be provided to the user via task-related UI elements (796). The user can then receive or download the configured game or application (797). The user can be the same user who interacted with the task in (791) or a different user.

[0220] Figure 7F This is a concrete example of a system using an autoencoder to implement user feedback as design feedback. As described above, the autoencoder can be used with a design library to create a set of latent factors that effectively represent a reduced set of features describing a given design. The system can access a design representation library, including shapes, images, graphics, data structures, etc. (A). The system can then generate a reconstructed design using the set of latent factors that effectively represent the set of features describing the design (B). The system can automatically create car designs based on raw data or evolved latent factors or improved design features (C). The system can receive results from a behavioral model created through a user-defined task (D). The system can use the behavioral model along with user classifications of the car designs in the form of semantic representations (e.g., sporty, family, compact) (E). The system uses a cost function-based scoring method (F) and performs optimization to create new feature vectors to automatically create design objects (G). The method completes if the optimization converges to a specific set of criteria (H).

[0221] Figure 8 This is a flowchart of an example process 800 for data quality improvement. In some implementations, process 800 may be executed by one or more systems. For example, process 800 may be performed by... Figure 1 -2 and Figure 4 The process 800 is implemented using a data quality processor 120, a DCDS 110, a user equipment 106, and a third party 140. In some embodiments, the process 800 may be implemented as instructions stored on a non-transitory computer-readable medium, and when executed by one or more servers, these instructions may cause one or more servers to perform the operations of the process 800.

[0222] Process 800 begins by receiving a request (802) from the user equipment for a digital component to be presented at the user equipment. For example, system 100 may receive request 108 from user equipment 106.

[0223] Process 800 continues by determining one or more attributes of the user based on one or more of the information provided by the user or contained in a request for digital components (804). As mentioned above... Figure 2A As discussed, data processor 124 can determine one or more attributes of a user based on information provided by the user, such as a user profile, or information contained in a request. For example, information provided in a user profile may include the user's age, gender, interests, location, and other characteristics. Information contained in request 108 may include anonymous information or information that cannot be used to identify the user, and may include, for example, the website that generated request 108 and the destination website the user navigated to. In one example, data processor 124 may determine that the user is a male aged 65 or older.

[0224] Process 800 continues by identifying behavioral models (806) corresponding to one or more attributes of the user. For example, model generator 126 may identify behavioral models that predict user behavior based on one or more attributes of the user. In one example, model generator 126 may identify a behavioral model for men aged 65 and over.

[0225] Process 800 continues by dynamically altering the presentation of the item depicted by the digital components (808) based on an identified behavioral model corresponding to one or more attributes of the user. For example, task processor 122 and / or data processor 124 may dynamically alter the presentation of the item depicted by the digital components, such as a task question, based on an identified behavioral model corresponding to one or more attributes of the user. In one example, task processor 122 may modify the task question about a mug design depicted in the task based on a behavioral model targeting men over 65 years of age.

[0226] In some implementations, task processor 122 selects the format in which feedback is requested for the digital components of a user's request for feedback on a project. For example, task processor 122 selects a format for a user's reaction to a particular mug design.

[0227] In some implementations, the data quality processor 120 dynamically alters the presentation of an item depicted by digital components based on identified behavioral models corresponding to one or more attributes of the user. This includes using machine learning or artificial intelligence techniques to identify feedback to be requested from the user regarding the item. For example, the data quality processor 120 may use the output of the model generator 126 to identify feedback to be requested from the user regarding the mug design.

[0228] In some implementations, task processor 122 verifies information requested by the digital components based on specific attributes corresponding to underrepresented segments of the user group. For example, task processor 122 verifies information requested by the task to be distributed based on the user's age attribute.

[0229] Process 800 continues by determining that the user has a specific attribute corresponding to an underrepresented segment of the user group in a database containing information about the item (810). For example, data processor 124 may determine that the segment of male users aged 65 and over is an underrepresented segment based on the number of threshold responses. In some embodiments, data processor 124 uses statistical analysis to identify underrepresented segments of the user group. Data processor 124 may then determine that the user of user device 106 has an age attribute corresponding to an underrepresented segment of the user group in a database containing information about the mug design.

[0230] Process 800 continues by generating digital components (812) in response to a request, including a presentation of dynamic changes to the item, soliciting feedback from the user about the item, and including a feedback mechanism that enables the user to submit feedback about the item, in response to a determination that the user has specific attributes corresponding to an underrepresented segment of the user group. For example, task processor 122 may generate a presentation of changes to the item or a task that solicits feedback from the user of user device 106 about the mug design and includes a feedback mechanism that enables the user to submit feedback about the item, such as a voting feature. In some implementations, DCDS 110 generates or selects digital components for distribution to user device 106, as described above. Figure 1 -7 describes it.

[0231] Process 800 continues by updating the database to include feedback from the user regarding the project (814). For example, data processor 124 can use feedback from the user on user device 106 to update the database containing information about the project.

[0232] In some implementations, in response to receiving feedback from users having specific attributes corresponding to an underrepresented segment of the user group, the data quality processor 120 tags the feedback information using one or more of the user's attributes and stores the tagged feedback information in a tagged, searchable database such as the tagged database 130.

[0233] Process 800 continues by modifying (816) the presentation of an item when it is distributed to other users who share one or more attributes of the user, based at least in part on feedback obtained from the user. For example, data quality processor 120 may modify the presentation when an item is distributed to other users who share one or more attributes of the user, based on feedback obtained from the user. This allows data quality processor 120 to adjust its tasks and product design based on the attributes of the users interacting with the system.

[0234] In some implementations, the data quality processor 120 can modify the presentation of an item when it is distributed to other users with one or more attributes of the user by selecting specific feedback mechanisms included in the digital components.

[0235] Figure 9 This is a flowchart of an example process 900 for automatically designing space exploration. In some implementations, process 900 may be executed by one or more systems. For example, process 900 may be performed by... Figure 1 -2 and Figure 4 The data quality processor 120, DCDS 110, user equipment 106, and third-party 140 are implemented. In some embodiments, process 600 may be implemented as instructions stored on a non-transitory computer-readable medium, and when executed by one or more servers, the instructions may cause one or more servers to perform the operations of process 900.

[0236] Process 900 begins by receiving a request (902) from the user equipment for a digital component to be presented at the user equipment. For example, as described above, system 100, and in particular DCDS 110, may receive a request 108 for a digital component to be presented at user equipment 106.

[0237] Process 900 continues by receiving a dataset (904) of user-provided information about a specific product design. For example, data quality processor 120 may receive user-provided responses regarding a specific product design, such as a handbag design. In some embodiments, the product design is specific to a particular product. The product design may be a service or a software product. For example, a specific product design may be the user interface design of a software application.

[0238] Process 900 continues by generating a visual representation (906) of design factors mapped to continuous shapes representing the geometry of potential product designs, based on a dataset of user-provided information. (See above regarding...) Figures 3-4 As described, model generator 126 can generate a design space that maps subjective factors to continuous shapes representing potential product designs. For example, model generator 126 can generate a design space that maps semantic factors such as descriptors to continuous shapes representing possible design systems.

[0239] The design space can be reversible, enabling the generation of visual representations that map design factors to continuous shapes representing potential product design geometry based on a dataset of user-provided information. This includes generating visual representations by mapping potential product design geometry to design factors.

[0240] Process 900 continues by segmenting the visual representation into multiple subdivisions based on design factor values ​​(908). As mentioned above... Figures 3-4 As described, the model generator 126 can divide the design space into groups based on the values ​​of subjective factors.

[0241] In some implementations, segmenting a visual representation into multiple subdivisions based on design factor values ​​includes segmenting the visual representation into multiple subdivisions such that each subdivision of the visual representation shares design factor values ​​within a defined range. For example, data processor 124 may segment a design space into multiple subdivisions based on a range of design factor values, such as a subjective rating of the perceived comfort level of a handbag design.

[0242] Process 900 continues by selecting a segment (910) of the visual representation containing fewer than a threshold amount of data points. (See above regarding...) Figures 3-4 As described, the data processor 124 can identify data segments based on metrics such as the number of data points at a threshold. For example, the data processor 124 can identify a segment of a very “fashionable” and very “practical” bag as having fewer data points than a threshold or failing to meet other metrics.

[0243] Process 900 continues by selecting the digital component from which information is requested from the user (912). For example, task processor 122 or DCDS 110 can select the digital component from which information is requested from the user. Task processor 122 can select or generate a task for requesting information from the user.

[0244] In some implementations, task processor 122 selects the format in which the request for information is responded to for the digital component requested from the user. For example, task processor 122 selects a format for the user's response to the design of a handbag.

[0245] In some implementations, selecting the format for requesting information includes choosing a specific feedback mechanism to provide along with dynamically changing digital components.

[0246] In some implementations, task processor 122 verifies information requested by the digital components based on specific attributes corresponding to underrepresented segments of the user group. For example, task processor 122 verifies information requested by the task to be distributed based on the user's age attribute.

[0247] Process 900 continues by dynamically altering the presentation of a digital component (914) based on a selected segmentation of visual representation, which requests information from the user about the segmentation of a visual representation containing fewer than a threshold amount of data points. For example, as mentioned above regarding... Figures 3-4 As described, task processor 122 can dynamically modify existing tasks or generate new tasks. In one example, task processor 122 can modify an existing task to present a new product design to the user that has not been previously generated or presented to the user.

[0248] In some implementations, dynamically changing the presentation of content items involves using machine learning or artificial intelligence techniques to specify the information to be requested by the digital components. For example, data quality processor 120 may use a machine learning model generated by model generator 126 to determine and specify the information to be requested by the task.

[0249] In some implementations, dynamically changing the presentation of a digital component includes determining, based on a request for the digital component presented at the user device, that the user of the user device is in a first user cluster interested in a particular product design; identifying user interface elements of the digital component based on this determination; and changing the user interface elements of the digital component's presentation. For example, data processor 124 may determine that the user of user device 106 is in a cluster of interested users, identify user interface elements of a task, and change these elements to customize the task for users already interested in handbag designs.

[0250] In some implementations, dynamically changing the presentation of a digital component includes determining, based on a request for the digital component presented on the user device, that the user of the user device is in a first user group interested in a particular product design, wherein the request for the digital component presented on the user device indicates one or more attributes of the user based on information provided by the user, identifying user interface elements of the digital component based on determining that the user of the user device is in a first user group interested in a particular product design, and changing the user interface elements presenting the digital component.

[0251] In some implementations, process 900 includes constructing a behavioral model based on feedback information to predict user acceptance of potential product design geometry. For example, model generator 126 may generate a behavioral model predicting user acceptance of a handbag product design. Modifications to design factors for a particular product design are based at least in part on the behavioral model.

[0252] Process 900 continues by distributing dynamically changed digital components for presentation at the user equipment (916). For example, DCDS 110 may distribute the task and any requested content as a response 114 to user equipment 106.

[0253] Process 900 continues by obtaining feedback information (918) from the user equipment via a feedback mechanism regarding the segmentation of the visual representation containing fewer than a threshold amount of data points. For example, DCDS 110 may receive response data 116 from user equipment 106 and provide response data 116 to data processor 124. As mentioned above... Figures 3-4 The data quality processor 120 can receive feedback from the user regarding specific subdivisions of the visual representation. For example, data processor 124 and DCDS 110 can receive feedback from user equipment 106 regarding design space subdivisions with fewer than a threshold number of data points.

[0254] In some implementations, the request for a digital component to be presented at the user equipment indicates user demographic information of the user of the user equipment. Process 900 may also include, based on the request for a digital component to be presented at the user equipment, identifying the user of the user equipment within a first user group, such as a female user group in California.

[0255] In some implementations, process 900 may further include receiving from a second user device a request for a digital component to be presented at the second user device, the digital component indicating user demographic information of the user of the second user device. System 100 (e.g., data processor 124) may then determine, based on the request for the digital component presented at the second user device, that the user of the second user device is in the same first user group as the user of the user device. For example, system 100 (e.g., data processor 124) may determine that the user of the second device is a woman from California. In response to determining that the user of the second user device is in the same first user group as the user of the user device, system 100 may provide a modified product design instead of a specific product design. For example, due to the similarity between the user of the first user device and the user of the second user device, task processor 122 may provide the user of the second user device with a modified handbag design instead of the original handbag design.

[0256] Process 900 continues by modifying design factors of a specific product design, at least in part, based on feedback obtained from the user, to create a modified product design (920). For example, model generator 126 may modify design factors of a handbag design, at least in part, based on feedback from the user, to create a modified handbag design.

[0257] As mentioned above Figures 3-4As described, model generator 126 can update the design space and / or behavioral model. For example, model generator 126 can update the dataset by providing feedback information as input to the training system of the behavioral model or design generator.

[0258] In some implementations, process 900 includes identifying, from a plurality of existing product designs, the one that most closely resembles the modified product design in terms of the maximum number of common design factor values. For example, data quality processor 120 may identify an existing product that most closely follows the modified design. For example, system 100 may modify an existing product and its manufacturing method instead of generating a completely new product. In some implementations, system 100 may provide the modified product design to an integrated manufacturing system. For example, data quality processor 120 may provide the modified product design to a 3D printing system or an automated manufacturing system for immediate production.

[0259] Figure 10 This is a block diagram of an example computer system 1000 that can be used to perform the operations described above. System 1000 includes a processor 1010, a memory 1020, a storage device 1030, and an input / output device 1040. Each of components 1010, 1020, 1030, and 1040 may be interconnected, for example, using a system bus 1050. Processor 1010 is capable of processing instructions for execution within system 1000. In one embodiment, processor 1010 is a single-threaded processor. In another embodiment, processor 1010 is a multi-threaded processor. Processor 1010 is capable of processing instructions stored in memory 1020 or storage device 1030.

[0260] The memory 1020 stores information within the system 1000. In one embodiment, the memory 1020 is a computer-readable medium. In one embodiment, the memory 1020 is a volatile memory cell. In another embodiment, the memory 1020 is a non-volatile memory cell.

[0261] Storage device 1030 provides high-capacity storage for system 1000. In one embodiment, storage device 1030 is a computer-readable medium. In various other embodiments, storage device 1030 may include, for example, a hard disk drive, an optical disk drive, a storage device shared by multiple computing devices (e.g., cloud storage devices) over a network, or some other high-capacity storage device.

[0262] Input / output device 1040 provides input / output operations for system 1000. In one embodiment, input / output device 1040 may include one or more network interface devices, such as Ethernet cards, serial communication devices, such as RS-232 ports, and / or wireless interface devices, such as 802.11 cards. In another embodiment, input / output device may include a driver device configured to receive input data and send output data to other input / output devices (e.g., keyboards, printers, and display device 1060). However, other embodiments may also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0263] Although already Figure 10 The example processing system described herein may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed herein and their equivalents, or one or more combinations thereof).

[0264] An electronic document (hereinafter referred to as a document for the sake of brevity) does not necessarily correspond to a file. A document can be stored as part of a file that contains other documents, as a single file dedicated to the document in question, or as multiple coordinating files.

[0265] Embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents, or one or more combinations thereof). Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a computer storage medium (or medium) for execution by or control of the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof, or may be contained in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. Computer storage media can also be one or more separate physical components or media, or contained in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0266] The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0267] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines used for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination thereof. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0268] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple coordinating files (e.g., files storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected through a communication network.

[0269] The processes and logic flows described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform actions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0270] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, or operatively coupled to receive data from or transfer data to one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by or integrated into dedicated logic circuits.

[0271] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user—such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor—and a keyboard and pointing device—such as a mouse or trackball—through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0272] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server; or middleware components, such as an application server; or front-end components, such as a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described in this specification; or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet) and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0273] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is created by computer programs running on their respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from that user). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0274] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the content that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0275] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments; rather, it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0276] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes described in the drawings do not necessarily require the specific order or sequence shown to obtain the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for improving robustness of a data set, performed by one or more data processing apparatus, comprising: receiving, from a user device, a request for a digital component for presentation at the user device; determining one or more attributes of the user based on one or more of information provided by the user or information contained in the request for a digital component; identifying a behavioral model corresponding to the one or more attributes of the user; dynamically altering presentation of an item depicted by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user; determining that the user has a particular attribute corresponding to an underrepresented segment of a population of users that provided feedback about the item included in a database containing information about the item based on a number of users with the particular attribute that previously provided feedback about the item; and in response to determining that the user has the particular attribute corresponding to the underrepresented segment of the population of users: generating, in response to the request, a digital component that includes the dynamically altered presentation of the item, solicits feedback about the item from the user, and includes a feedback mechanism that enables the user to submit feedback about the item; updating the database to include the feedback about the item obtained from the user; and modifying presentation of the item when distributed to other users having the one or more attributes of the user based at least in part on feedback obtained from the user.

2. The method of claim 1, further comprising: in response to receiving the feedback from the user having the particular attribute corresponding to the underrepresented segment of a population of users, tagging the feedback with the one or more attributes of the user; and storing the tagged feedback in a tagged, searchable database. Determining that the user has a particular attribute corresponding to an underrepresented segment of a population of users that provided feedback about the item included in the database includes using statistical analysis to identify the underrepresented segment of the population of users. Modifying presentation of the item when distributed to other users having the one or more attributes of the user based at least in part on the feedback obtained from the user includes selecting a particular feedback mechanism included by the digital component.

3. The method of claim 1, wherein, Dynamically altering presentation of an item depicted by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user includes using machine learning or artificial intelligence techniques to identify feedback about the item to solicit from the user.

4. The method of claim 1, wherein, Determining one or more attributes of the user is based on information provided by the user, and 5. The method of claim 1, wherein, wherein dynamically altering presentation of an item depicted by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user includes updating the identified behavioral model based on the information provided by the user.

6. The method of claim 1, wherein, ​ ​ 7. The method of claim 1, wherein, determining the one or more attributes of the user is based on information included in the request for the digital component, and wherein dynamically altering the presentation of the item depicted by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user includes updating the identified behavioral model based on the information included in the request for the digital component.

8. The method of claim 1, further comprising: for the digital component soliciting feedback from the user regarding the item, selecting a format in which the feedback is solicited; and verifying the information solicited by the digital component based on the particular attribute corresponding to the underrepresented segment of the user population.

9. The method of claim 1, wherein, the digital component displays two different product design shapes for a shoe, 10. The method of claim 1, wherein, wherein the feedback obtained from the user regarding the item includes a selection of one of the two different product design shapes for the shoe that the user believes pairs better with the outfit. the digital component displays two different product design shapes for a car, wherein the digital component specifies a particular subjective product design shape descriptor, wherein the feedback mechanism is a slider, and 11. The method of claim 1, wherein, wherein the feedback obtained from the user regarding the item includes a selection of one of the two different product design shapes for the car that the user believes can be better described by the particular subjective product design shape descriptor. the digital component displays three or more different product design shapes for three or more different item types, wherein the feedback obtained from the user regarding the item includes a selection of two or more different product design shapes that the user believes are visually harmonious, and 12. The method of claim 1, wherein, wherein the method further comprises using the feedback obtained from the user in a separate model. the digital component asks the user to create a new product design, wherein the feedback mechanism receives user input indicative of a new product design, and 13. The method of claim 1, wherein, wherein the method further comprises providing the user with a product having the new product design. the digital component asks the user to modify an existing product design to produce a customized product design, wherein the feedback mechanism receives user input modifying one or more aspects of the existing product design, and 14. The method of claim 1, wherein, wherein the method further comprises providing the user with a product having the customized product design. the digital component displays two different software attributes, wherein the feedback obtained from the user regarding the item includes a selection of one of the two different software attributes that the user prefers, and 15. The method of claim 1, wherein, wherein the method further comprises providing the user with a software package having the selected one of the two different software attributes that the user prefers. the underrepresented segment of the user population is determined based on a number of users in the user population providing feedback that have the particular attribute and a second number of users in the user population providing feedback that do not have the particular attribute but have a different attribute.

16. A system comprising: one or more processors; and one or more memory elements comprising instructions that, when executed, cause the one or more processors to perform operations comprising: receiving, from a user device, a request for a digital component for presentation at the user device; determining one or more attributes of the user based on one or more of information provided by the user or information included in the request for a digital component; identifying a behavioral model corresponding to the one or more attributes of the user; dynamically altering a presentation of an item depicted by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user; determining that the user has a particular attribute corresponding to an underrepresented segment of a population of users that provided feedback about the item included in a database containing information about the item based on a number of users with the particular attribute that previously provided feedback about the item; and in response to determining that the user has the particular attribute corresponding to the underrepresented segment of the population of users: generating, in response to the request, a digital component that includes the dynamically altered presentation of the item, solicits feedback about the item from the user, and includes a feedback mechanism that enables the user to submit feedback about the item; updating the database to include the feedback about the item obtained from the user; and modifying a presentation of the item when distributed to other users having the one or more attributes of the user based at least in part on the feedback obtained from the user.

17. The system of claim 16, the operations further comprising: in response to receiving the feedback from the user having the particular attribute corresponding to the underrepresented segment of the population of users, tagging the feedback with the one or more attributes of the user; and storing the tagged feedback in a tagged, searchable database. Determining that the user has a particular attribute corresponding to an underrepresented segment of a population of users that provided feedback about the item included in the database includes using statistical analysis to identify the underrepresented segment of the population of users.

18. The system of claim 16, wherein, Modifying a presentation of the item when distributed to other users having the one or more attributes of the user based at least in part on the feedback obtained from the user includes selecting a particular feedback mechanism included by the digital component.

19. The system of claim 16, wherein, Dynamically altering a presentation of an item depicted by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user includes using machine learning or artificial intelligence techniques to identify feedback about the item to solicit from the user.

20. The system of any one of claims 16 to 19, wherein, 21. A non-transitory computer storage medium encoded with instructions that, when executed by a distributed computing system, cause the distributed computing system to perform operations comprising: ​ receiving, from a user device, a request for a digital component for presentation at the user device; determining one or more attributes of the user based on one or more of the information provided by the user or information included in the request for the digital component; identifying a behavioral model corresponding to the one or more attributes of the user; dynamically altering a presentation of an item depicted by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user; determining that the user has a particular attribute corresponding to a under-represented segment of a population of users that provided feedback about the item included in a database containing information about the item based on a number of users with the particular attribute that previously provided feedback for the item; and in response to determining that the user has the particular attribute corresponding to the under-represented segment of the population of users: in response to the request, generating a digital component that includes the dynamically altered presentation of the item, solicits feedback about the item from the user, and includes a feedback mechanism that enables the user to submit feedback about the item; updating the database to include the feedback about the item obtained from the user; and modifying a presentation of the item when distributed to other users having one or more attributes of the user based at least in part on the feedback obtained from the user.

22. A computer storage medium encoded with instructions that, when executed by a distributed computing system, cause the distributed computing system to perform operations comprising the method of any of claims 2-15.

Images