Automatically and intelligently exploring design space

Abstract

Methods, systems, and computer-readable media include receiving requests for digital components from a user device; receiving a dataset of user-provided information about a particular product design; generating a visual representation based on the dataset that maps design factors to the geometry of a potential product design; segmenting the visual representation based on design factor values; selecting segments containing fewer than a threshold number of data points; selecting digital components; dynamically changing the presentation of the digital components from which information about the segments is solicited from the user based on the selected segments; distributing the dynamically changed digital components for presentation at the user device; obtaining feedback information from the user device about the segments containing fewer than a threshold number of data points via a feedback mechanism; and modifying design factors of a particular product design at least in part based on the feedback information.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefits of U.S. Application No. 16 / 943,126, filed July 30, 2020, and U.S. Provisional Application No. 63 / 010,438, filed April 15, 2020, the contents of which are incorporated herein by reference. Technical Field

[0003] This document relates to providing a data collection and model generation process that continuously reduces bias introduced by using data points from unevenly distributed or unrepresentative populations and explores the design space. Summary of the Invention

[0004] Generally, an innovative aspect of the subject matter described in this specification can be embodied in a method for generating a new product design, the method comprising receiving from a user device a request for digital components presented on the user device; receiving a dataset of user-provided information about a particular product design; generating a visual representation based on the dataset of user-provided information that maps design factors to a continuous shape representing the geometry of a potential product design; segmenting the visual representation into multiple segments based on design factor values; selecting segments of the visual representation containing fewer than a threshold number of data points; selecting digital components for which information is solicited from the user; dynamically changing the presentation of the digital components for which information is solicited from the user about the segments of the visual representation containing fewer than a threshold number of data points based on the selected segments of the visual representation; distributing the dynamically changed digital components for presentation at the user device; obtaining feedback information from the user device and through a feedback mechanism regarding the segments of the visual representation containing fewer than a threshold number of data points; and modifying design factors of a particular product design, at least in part, based on the feedback information obtained from the user, to create a modified product design.

[0005] These and other embodiments may each optionally include one or more of the following features.

[0006] In some implementations, the method includes selecting a format for soliciting information for a specific content item from a user, and verifying the information solicited for the specific content item based on the selected fragment of visual representation. In some implementations, selecting the format for soliciting information includes selecting specific feedback mechanisms provided by dynamically changing digital components.

[0007] In some implementations, the method includes determining, based on a request for a digital component presented at a user equipment, that the user of the user equipment is in a first user group, wherein the request for the digital component presented at the user equipment indicates user demographic information of the user of the user equipment. The method can also include receiving from a second user equipment a request for a digital component for presentation at the second user equipment indicating user demographic information of the user of the second user equipment; determining, based on the request for the digital component presented at the second user equipment, that the user of the second user equipment is in the same first user group as the user of the user equipment; and, in response to determining that the user of the second user equipment is in the same first user group as the user of the user equipment, providing a modified product design instead of a specific product design.

[0008] In some implementations, segmenting a visual representation into multiple segments based on design factor values ​​includes dividing the visual representation into multiple segments based on design factor values, such that each segment of the visual representation shares design factor values ​​within a defined value range.

[0009] User interface elements can be, for example, images of a product or brand, audio or video representations, feedback mechanisms, the theme of the task, and / or the wording of the task. For instance, data quality processor 120 can determine that the user of user device 106 belongs to a user group interested in the product design of a handbag and identify the visual theme of the task to be changed. In some embodiments, data processor 120 can change user elements by replacing them. In some embodiments, data processor 120 can change user elements by modifying them. For example, data quality processor 120 can change the visual theme of the task by adding elements specific to a particular product design that the user is identified as being of interest, including the logo of a particular brand of handbag, a color scheme reminiscent of a particular brand of handbag, an audio track used with a particular brand of handbag, and other elements.

[0010] In some implementations, dynamically changing the presentation of a digital component includes determining, based on a request for the digital component to be presented at the user device, that the user of the user device is in a first user group, wherein the request for the digital component to be presented at the user device includes information indicating one or more attributes of the user, identifying user interface elements of the digital component based on determining that the user of the user device is in the first user group, and changing the user interface elements of the digital component presentation.

[0011] In some implementations, mapping design factors to a visual representation of a continuous shape representing a potential product design geometry is reversible, such that generating a visual representation of a design factor to a continuous shape representing a potential product design geometry based on a dataset of user-provided information includes generating a visual representation by mapping the potential product design geometry to design factors.

[0012] In some implementations, the method includes identifying the closest existing product design from a plurality of existing product designs based on the modified product design, the existing product design having a plurality of design factor values ​​common to the modified product design.

[0013] In some implementations, the method includes providing a modified product design to an integrated manufacturing system.

[0014] In some implementations, the method includes constructing a behavioral model based on feedback information to predict user acceptance of potential product design geometry, wherein design factors that modify a particular product design are at least partially based on the behavioral model.

[0015] In some implementations, specific product design refers to the user interface design of a software application.

[0016] In some implementations, dynamically changing the presentation of content items involves using machine learning or artificial intelligence techniques to specify information requested by the digital components.

[0017] Other embodiments of this aspect include corresponding systems, apparatuses, and computer programs configured to perform the actions of the method and encoded on a computer storage device.

[0018] Specific embodiments of the subject matter described in this document may be implemented to achieve one or more of the following advantages. In certain environments, there has been no prior method for automatically and systematically reducing bias in datasets that are not representative of the group or are identified as lacking data in particular segments, and this drawback is addressed by the techniques, devices, and systems discussed herein.

[0019] In some implementations of this new system, underrepresented segments within the group for which data is collected are identified, and tasks are generated and distributed to users within these segments. These tasks solicit responses from users on specific topics or domains, allowing these responses to supplement the existing dataset. For example, the system can determine that responses from users within a specific age range regarding a preferred choice between two products are less than a threshold amount, and then generate a task to distribute to users within that age range, requiring them to choose between the two products. These responses are received and processed by the system as a supplement to the existing dataset, thereby improving data quality. The system continuously monitors the dataset, providing a solution that automatically maintains data quality even when the dataset is updated.

[0020] This new system accesses sophisticated data processing infrastructure that cleans, processes, and maintains a comprehensive set of labeled, searchable data that can be used in many different situations to improve model results. Previously, no solution could automatically improve and maintain data quality as more data was collected. If a model uses an incomplete or unrepresentative dataset, it can produce results that do not represent the actual behavior of the population. The new system automatically supplements the dataset from which the model draws data, thereby improving the robustness of the data and the accuracy of the model results. By automatically identifying insufficient or unrepresentative fragments of the population, the system reduces the bias of datasets used as input to various models. These more robust datasets, in turn, improve the reliability and accuracy of the results from models that rely on this data. The improved datasets can be labeled and used by various parties, including content providers and product manufacturers who may lack the infrastructure to maintain labeled and searchable datasets or who do not have access to such datasets.

[0021] In addition to improving the dataset, the system can automatically explore the design space. The design space is a conceptual representation of design values ​​and can be referred to as a continuous shape. The design space can be associated with a specific product or service and can be multidimensional, representing possible design values ​​for parameters of interest. In some implementations, the design space can be expanded to map design parameters to semantic values. For example, the system can create models using mappings of a product's semantic attributes and geometric features; these models exist in a single continuous shape space that represents a range of possible attribute values. Designs can be based on one or more sensory features. For example, a design can be visual, auditory, tactile, odor-based, or taste-based. For example, the design of a car seat can include both visual and tactile features.

[0022] This new system automatically explores the design space by identifying skewnesses with little or no data and generating tasks about these identified fragments to be distributed to users. By collecting data on these fragments, the system allows for the inclusion of previously unexplored designs. For example, the system can automatically generate designs with parameter values ​​that have not yet been presented to users for feedback. Based on received feedback, the system can continuously update existing designs and generate new designs, soliciting user feedback on the new designs before prototyping, manufacturing, and distribution phases. This feedback can be used as input to outputs such as models predicting user preferences. The system can determine the direction guiding product design based on the output of data analysis and behavioral models, allowing content providers, product designers, and manufacturers to focus on designs most likely to be accepted by the target consumer group. In other words, the improved update process reduces the number of feedback cycles required to complete a product design, thereby reducing the computational resources required throughout the entire design cycle from initial data collection to design completion.

[0023] This approach enables rapid product design and development, thereby more accurately and reliably meeting the needs and expectations of diverse consumer groups. Furthermore, the improved update process enhances the efficiency of the design system by improving data quality and reducing the number of feedback cycles required to collect user data for input into behavioral models. This system provides product designers, developers, and manufacturers with a way to rapidly receive diverse feedback globally and determine user preferences based on demographics before the manufacturing and shipping stages, which typically requires significant resource investment. This approach can allow for personalized design and manufacturing for individuals or user groups. For example, manufacturers can use this method to examine individual preferences to measure acceptance statistics for a larger market.

[0024] The technology described in this document enables the system to generate high-quality data relevant to a specific product or service using fewer resources and performing fewer operations. By automatically detecting and reducing biases in the dataset, the system allows models using the data to provide more accurate and reliable results. Furthermore, the system reduces the amount of resources required to complete the design of a new product or service by allowing for targeted exploration of the design space and continuous improvement of data quality.

[0025] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0026] Figure 1 This is a block diagram of an example environment used for data quality improvement and design space exploration.

[0027] Figure 2AAn example data flow of the data quality improvement process is shown.

[0028] Figure 2B and Figure 2C The model training process is described.

[0029] Figure 3 An example data flow of the data quality improvement process is shown.

[0030] Figure 4 An example data stream for designing a space exploration process is shown.

[0031] Figure 5A and Figure 5B The data flow for a specific example is described, where the system uses a behavioral model to generate tasks for the user.

[0032] Figure 6A and Figure 6B The data flow is described as a specific example of how the system integrates user feedback into the design cycle.

[0033] Figure 7A and Figure 7B The data flow is described in a specific example where the system implements user feedback to customize existing designs and products.

[0034] Figures 7C to 7F It describes a specific example of how the system incorporates user feedback into the design cycle.

[0035] Figure 8 This is a flowchart of an example process for improving data quality.

[0036] Figure 9 This is a flowchart of an example process for automatically designing space exploration.

[0037] Figure 10 This is a block diagram of an example computing system.

[0038] The same reference numerals and names in different figures denote the same elements. Detailed Implementation

[0039] This paper describes methods, systems, and devices for improving data quality, reducing inherent bias, and enabling automated, intelligent design of spatial exploration. The accuracy and representativeness of a model depend on the input data provided to it. The proposed system improves the quality of data available for modeling and product development across various systems. It enables the collection of user feedback and behavioral data through various methods.

[0040] In some implementations, the system generates tasks to be distributed to users. Each task can be a question, assignment, or other form of user input solicitation. User input can be tagged and used as part of a comprehensive database that can be used in different contexts. For example, a task can be presented to the user where they must choose the single logo design that appears most appealing from multiple logo designs. The user's selection can then be tagged using the user's demographic information and stored as part of a searchable database of tags. The system can determine, based on analysis of the tagged dataset, whether data from a specific user demographic or about a specific product segment is missing, insufficient, or not representative of a known population, and automatically generate tasks or questions to collect more data and reduce inherent biases in the unrepresentative dataset. Supplementary tagged datasets can be provided as input to various models. For example, the tagged dataset can be provided to behavioral models to predict whether a particular design will appeal to a specific user group. Various models can be used to predict user reactions and acceptance, for example, of a particular design.

[0041] The system also allows for the automated and intelligent exploration of specific design spaces. For example, based on analysis of labeled datasets, the system can identify areas of the design space where data is missing, insufficient, or not representative of known populations, and automatically generate tasks or questions to gather more data. The system can explore the design space by generating product designs based on unexplored areas using artificial intelligence and machine learning models. These artificially generated designs can then be presented to users along with tasks that solicit feedback.

[0042] Figure 1 This is a block diagram of an example environment 100 for data quality improvement and design space exploration. Example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. Network 102 connects an electronic document server 104 (“Electronic Document Server”), user equipment 106, and a digital component distribution system 110 (also referred to as DCDS 110). Example environment 100 may include many different electronic document servers 104 and user equipment 106.

[0043] User equipment 106 is an electronic device capable of requesting and receiving resources (e.g., electronic documents) via network 102. Example user equipment 106 includes personal computers, mobile communication devices, and other devices capable of sending and receiving data via network 102. User equipment 106 typically includes user applications, such as web browsers, to facilitate the sending and receiving of data via network 102; however, local applications executed by user equipment 106 may also facilitate the sending and receiving of data via network 102.

[0044] One or more third parties 140 include content providers, product designers, product manufacturers, and other parties involved in the design, development, manufacture, marketing, or distribution of a product or service.

[0045] An electronic document is data that presents a set of content at user device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Local applications (e.g., “applications”), such as those installed on mobile, tablet, or desktop computing devices, are also examples of electronic documents. Electronic document 105 (“electronic document”) can be provided to user device 106 by electronic document server 104. For example, electronic document server 104 can include a server hosting a publisher’s website. In this example, user device 106 can initiate a request for a given publisher’s web page, and electronic document server 104 hosting the given publisher’s web page can respond to the request by sending machine hypertext markup language (HTML) code that initiates the rendering of the given web page at user device 106.

[0046] Electronic documents can include a variety of content. For example, electronic document 105 can include static content (e.g., text or other specified content) that is inherent to the electronic document itself and / or does not change over time. Electronic documents can also include dynamic content that can change over time or based on each request. For example, the publisher of a given electronic document can maintain a data source for populating portions of the electronic document. In this example, a given electronic document can include tags or scripts that, when user device 106 processes (e.g., renders or executes) the given electronic document, cause user device 106 to request content from the data source. User device 106 integrates the content obtained from the data source into the rendering of the given electronic document to create a composite electronic document that includes content obtained from the data source.

[0047] In some cases, a given electronic document may include a digital content tag or digital content script referencing DCDS 110. In these cases, when user equipment 106 processes the given electronic document, user equipment 106 executes the digital content tag or digital content script. Execution of the digital content tag or digital content script configures user equipment 106 to generate a request 108 for digital content, which is transmitted to DCDS 110 via network 102. For example, the digital content tag or digital content script enables user equipment 106 to generate a packetized data request including header and payload data. Request 108 may include data such as the name (or network location) of the server for the requested digital content, the name (or network location) of the requesting device (e.g., user equipment 106), and / or information that DCDS 110 can use to select the digital content offered in response to the request. User equipment 106 transmits request 108 to the server of DCDS 110 via network 102 (e.g., a telecommunications network).

[0048] Request 108 can include data specifying the characteristics of electronic documents and locations where digital content can be presented. For example, data specifying references to electronic documents (e.g., web pages) that will present digital content (e.g., URLs), available locations in the electronic document that can be used to present digital content (e.g., digital content slots), the size of the available locations, the position of the available locations in the presentation of the electronic document, and / or the media types eligible to be presented in these locations can be provided to DCDS 110. Similarly, data specifying keywords (“document keywords”) assigned to select electronic documents or entities (e.g., people, places, or things) referenced by the electronic documents can also be included in Request 108 (e.g., as payload data) and provided to DCDS 110 to facilitate the identification of digital content items eligible to be presented with electronic documents.

[0049] Request 108 may also include data related to other information, such as information already provided by the user, geographic information indicating the state or region from which the request was submitted, or other information that provides context for the environment in which the digital content will be displayed (e.g., the type of device that will display the digital content, such as a mobile device or a tablet). The information provided by the user may include demographic data of the user of user device 106. For example, demographic information may include characteristics such as age, gender, geographic location, education level, marital status, household income, occupation, hobbies, social media data, and whether the user owns specific projects.

[0050] For the situations discussed here where systems collect or may use personal information about users, users may be given the opportunity to control whether a program or feature collects personal information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current location), or to control whether and / or how content that may be more relevant to the user is received from the content server. Furthermore, certain data may be anonymized in one or more ways before being stored or used, thereby removing personally identifiable information. For example, a user's identity may be anonymized, making it impossible to determine the user's identity, or, where location information is available, the user's geographic location may be generalized (e.g., city, zip code, or state), making it impossible to determine the user's specific location. Therefore, users can control how content servers collect and use information about them.

[0051] Data specifying the characteristics of user equipment 106 can also be provided in request 108, such as information identifying the model of user equipment 106, the configuration of user equipment 106, or the size (e.g., physical size or resolution) of the electronic display (e.g., a touchscreen or desktop monitor) displaying the electronic document. Request 108 can be transmitted, for example, over a packet network, and request 108 itself can be formatted as packet data with a header and payload data. The header can specify the destination of the packet, and the payload data can include any of the information discussed above.

[0052] In response to receiving request 108 and / or using the information included in request 108, DCDS 110 selects digital content to be presented with a given electronic document. In some implementations, DCDS 110 is implemented in a distributed computing system (or environment) including, for example, a server and a plurality of interconnected computing devices that identify and distribute digital content in response to request 108. This plurality of computing devices operates together to identify a set of digital content eligible for presentation in an electronic document from a corpus of millions or more available digital content. For example, millions or more of available digital content can be indexed in a digital components database 112. Each digital content index entry can reference the corresponding digital content and / or include distribution parameters (e.g., selection criteria) that regulate the distribution of the corresponding digital content.

[0053] In some implementations, the digital components from the digital component database 112 can include content provided by a third party 140. For example, the digital component database 112 can receive photographs of public intersections from a third party 140 that uses machine learning and / or artificial intelligence to navigate public streets. In another example, the digital component database 112 can receive specific questions from a third party 140 that provides services to cyclists, and the third party 140 expects a response from the user.

[0054] The identification of qualified digital content can be segmented into multiple tasks, and then these tasks can be distributed among computing devices in the group of multiple computing devices. For example, different computing devices in the group of multiple computing devices can each analyze different parts of the digital component database 112 to identify various digital content with distribution parameters that match the information included in request 108.

[0055] DCDS 110 aggregates results received from the group of multiple computing devices and uses information associated with the aggregated results to select one or more instances of digital content to be provided in response to request 108. In turn, DCDS 110 can generate and transmit response data 114 (e.g., digital data representing a response) via network 102, enabling user device 106 to integrate the selected set of digital content into a given electronic document, such that the selected set of digital content and the content of the electronic document are presented together on the display of user device 106. The digital content distributed by DCDS 110 and represented by response data 114 can include, for example, digital content solicited from the user. This input can be analyzed, tagged, and stored as part of a comprehensive database, such as tag database 130. Tag database 130 stores tagged data that has been analyzed and categorized. Tag database 130 can be searched and can store user-related data, including user demographics, user response data, and other user characteristics. For example, tag database 130 can store anonymous user demographics and associate them with the user's response to a question previously presented to the user. User input is transmitted as response data 116 from user equipment 106 to data quality processor 120.

[0056] Data quality processor 120 generates digital content soliciting user input, receives and processes user input data, and generates and modifies design spaces and designs. Data quality processor 120 includes task processor 122, data processor 124, and model generator 126. Task processor 122 generates tasks to be distributed to users. Data processor 124 analyzes and tags the input received in response to the tasks. Model generator 126 generates and modifies design spaces and designs based on tagged data and input from third parties such as content providers, product designers, and manufacturers. For ease of explanation, task processor 122, data processor 124, and model generator 126 are... Figure 1The data quality processor 120 is shown as a separate component of the data quality processor 120. The data quality processor 120 can be implemented as a single system on a computer-readable medium, which may be non-transitory. In some embodiments, one or more of the task processor 122, data processor 124, and model generator 126 can be implemented as an integrated component of a single system.

[0057] Task processor 122 creates digital content that solicits input from a user or task. Task processor 122 communicates with DCDS 110, electronic document server 104, and third party 140. Data quality processor 120 is capable of collecting data from tasks directly issued by task processor 122 or from tasks issued by a third party, such as third party 140, which provides access to its data source to data quality processor 120. Tasks can include content requiring different levels of interaction, from activities asking a user to draw a picture to questions only requiring the user to select an answer, to click activities requiring the user to grant the system access to user data. In some implementations, tasks can include questions requesting the user to answer input. For example, a task presented to a user can include the question "Do you prefer chocolate or vanilla ice cream?" and the user can enter their answer or choose from pre-selected answers. In some implementations, tasks can include activities requiring more user participation. For example, a task presented to a user can ask the user to select one or more portions of an image including a traffic intersection with bicycles, and the user can click or otherwise indicate the appropriate portion. In some implementations, tasks can include authentication protocol challenges, such as CAPTCHA or reCAPTCHA.

[0058] In addition to generating tasks for users, task processor 122 can also modify tasks. For example, task processor 122 can modify tasks that have previously been provided to one or more users, and modify the task to collect different data, ask more targeted questions, or otherwise change the direction of the task. Task processor 122 and its output will be described in further detail below.

[0059] Data processor 124 receives and processes data to identify missing, inaccurate, underrepresented, or unrepresentative data, and automatically determines data quality improvement solutions. Data processor 124 analyzes specific datasets and determines whether existing data meets quality thresholds based on design guidelines and other inputs. Data processor 124 is capable of processing response data received from user devices and existing response data. For example, data processor 124 can determine whether existing user response data stored in the tag database 130 for a specific camping backpack design includes a representative number of responses from consumers of that age group by determining the ratio of responses received from consumers aged 45 to 54 to responses received from consumers of other ages and comparing the existing ratio to the expected or actual ratio in the camping backpack target market. Data processor 124 is also capable of determining whether design values ​​within the design space have been explored, or whether there is sufficient data about these values. For example, data processor 124 can determine whether a trackpad of a specific size on a laptop has received a sufficient number of user responses by comparing the existing number of user responses to a threshold number of user responses.

[0060] Data processor 124 is also capable of receiving and extracting data collected during the digital content distribution process. For example, data analyzer 124 is capable of receiving request data 108 and response data 114 to determine the group and characteristics of users represented by cookies indicated in request data 108 and response data 114. Data analyzer 124 is capable of storing demographic and other characteristic data in a database, such as tag database 130. In some embodiments, data analyzer 124 is capable of retrieving data from tag database 130 that has already been analyzed and tagged by other systems. Data analyzer 124 is capable, for example, of retrieving data from tag database 130 that indicates demographic data of the users who provided request data 108 and received response data 114. Data processor 124 is capable of segmenting data, for example, based on user demographic information. Data processor 124 and its output will be described in further detail below.

[0061] In addition to the descriptions above, users may be given control over whether and when the systems, programs, or features described herein can collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current location), and whether to send content or communications to the user from the server. Furthermore, some data may be processed in one or more ways before being stored or used to remove personally identifiable information. For example, a user's identity may be processed so that the user's personally identifiable information cannot be determined, or, where location information is available, the user's geographic location may be generalized (e.g., city, zip code, or state), thus preventing the determination of the user's specific location. Therefore, users have control over what information about themselves is collected, how that information is used, and what information is provided to them.

[0062] Model generator 126 generates, modifies, and maintains design spaces and models. Model generator 126 can generate design spaces and / or models based on data provided by data processor 124 or retrieved from the tag database 130. For example, model generator 126 can generate behavioral models predicting user preferences for a particular design based on data provided by data processor 124. These behavioral models map subjective factors to continuous semantic shapes as functions of demographic information, creating a design space that can be used to optimize the design under specific constraints.

[0063] The output of this model ranges from a specific design to a predicted user response to that design. The model generator 126 and its output will be described in further detail below.

[0064] The techniques described below enable the system to continuously and automatically improve data quality and explore the design space.

[0065] Figure 2A It shows Figure 1 Example data flow 200 is an example of a data quality improvement process in an example environment. The operation of data flow 200 is performed by various components of system 100. For example, the operation of data flow 200 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tag database 130.

[0066] The process begins in step A, where data processor 124 of data quality processor 120 analyzes and segments a dataset associated with a user group. In some implementations, the dataset is existing data retrieved from a labeled database 130. For example, the dataset could include a set of user inputs in response to a reCAPTCHA that asks a user to select all squares from a grid of photos that show a portion of a vehicle such as a bicycle. In another example, the dataset could include a set of user inputs in response to a question asking users how much they typically spend on dog food each month. User inputs could be associated with user characteristic data of the user providing the input. For example, user characteristic data could include demographic data and browsing history, as well as other data that system 100 can access and has permission to use. In another example, the dataset could include a set of user inputs in response to a question that asks users to rate their likelihood of purchasing a particular handbag design within a range of subjective descriptors such as “practical” or “fashionable.”

[0067] Data processor 124 segments the data based on various parameters, including user characteristic data. For example, data processor 124 can segment the dataset into fragments based on user age, user location, and / or user interests, as well as other user characteristic data that system 100 can access and has permission to use. In another example, data processor 124 can segment the data based on characteristics of the data itself. For example, data processor 124 can segment the dataset based on the values ​​of specific subjective factors related to product design, such as perceiving the "fashionability" of a handbag product design based on user feedback.

[0068] The process continues to step B, where data processor 124 identifies inadequate segments of the dataset based on one or more metrics. In some implementations, this metric is provided by a third party 140, such as a product designer. For example, the metric could be the number of user responses and the target demographic information of the users who responded. In some implementations, the metric is determined automatically. For example, the metric could be a threshold difference in the proportion of responding users, where a dataset with a proportion difference greater than the threshold difference can be considered not representative of the actual group soliciting responses. In one example, data processor 124 can identify a bicycle detection segment in the vehicle detection set as insufficiently granular.

[0069] The process continues to step C, where task processor 122 dynamically modifies the task to be presented to the user based on the identified fragment. In some embodiments, task processor 122 dynamically modifies existing tasks that have been previously generated and / or presented to the user. In some embodiments, task processor 122 generates entirely new tasks to be presented to the user. Task processor 122 is capable of performing this modification in real time in response to the identification of fragments. For example, data quality processor 120 is capable of continuously monitoring the quality of the dataset and updating its metrics based on new and updated information received.

[0070] In one example, task processor 122 automatically alters one aspect of a previously distributed task that required the user to select all squares containing a portion of a bicycle by modifying the grid system to provide better resolution using smaller squares. In step B, data processor 124 automatically determines that a larger granularity is needed, and using that information, task processor 122 is able to divide the intersection photograph containing the bicycle into smaller squares.

[0071] Data quality processor 120 automatically modifies or generates new tasks based on analysis of existing datasets and performs additional operations that contribute to task generation. For example, data quality processor 120 can determine that the modified task includes providing the user with photographic data of intersections where bicycles are visible. Data processor 122 can receive specific photographic data of intersections from a third party 140. Data processor 122 can also automatically acquire data to be provided as part of the task. For example, data processor 122 can retrieve photographic data of public intersections marked as having at least one bicycle in the field of view from, for example, a tagging database 130. Data processor 122 can then perform data cleaning operations, including erasing personally identifiable information, cleaning data, and adjusting data to make it usable, as well as other operations. For example, data processor 122 can adjust the stream of live photographs from a street camera positioned at a public intersection by filtering out images that do not contain bicycles, adjusting lighting, and creating a greater dynamic range in the images. Data processor 122 can perform complex data processing operations, including removing objects that obstruct another object and enhancing the focus of a specific object, as well as other operations.

[0072] The task processor 122 can also determine one or more distribution parameters that must be met before distributing the task based on the identified fragments. Distribution parameters can include user characteristics that a user must possess in order to accept the task. For example, distribution parameters can include specific demographic information about women aged 18 to 24 living on the West Coast of the United States.

[0073] Task processor 122 can modify tasks to, for example, sample existing or new areas of space within the tag database 130 for images, videos, or audio objects. Task processor 122 can also test the removal or addition of brand information to assess, for example, user reactions to the brand or user bias.

[0074] The process continues to step D, in which task processor 122 transmits dynamically changed or generated tasks to DCDS 110 for distribution to users. For example, task processor 122 can transmit task data indicating the tasks to be presented to the user and the content to be presented to the user as part of the tasks. Task processor 122 can include distribution parameters that must be met in order to distribute tasks to specific users. For example, task processor 122 can include demographic data of the target user to whom the tasks can be presented.

[0075] The process continues to step E, in which DCDS 110 receives a request 108 for content from user equipment 106. Request 108 is transmitted from user equipment 106 to DCDS 110 when the client device interacts with digital content. For example, if a user of user equipment 106 clicks a link to download a shopping app, the link enables user equipment 106 to transmit request 108 to DCDS 110. Request 108 may include interaction tracking data from client equipment 106. For example, request 108 may include tracking data such as indications of the interaction, the digital content that user equipment 106 interacted with, and an identifier that uniquely identifies user equipment 106. In some implementations, request 108 includes indications of the digital content provider and the location of the destination server hosting the requested resource.

[0076] The process continues to step F, in which DCDS 110 transmits response data 114 to user equipment 106. As described above, response data 114 indicates tasks to be distributed to the user that satisfy specific distribution parameters, in addition to the requested electronic document. In response to DCDS 110 receiving request 108 and determining that the distribution parameters are satisfied based on the received distribution parameters and the user data indicated in request 108, DCDS 110 transmits response data 114 to user equipment 106. For example, DCDS 110 can determine, based on the received request data 108, that the user of user equipment 106 is a 22-year-old female residing in Oregon, and therefore the distribution parameters are satisfied. DCDS 110 can then transmit the requested electronic document and dynamically changed tasks to user equipment 106 in the form of response data 114.

[0077] The process continues to step G, in which DCDS 110 receives response data 116 from user device 106. In response to the user on user device 106 completing the task provided in response data 114, user device 106 transmits response data 116 to DCDS 110. Response data 116 includes user information such as demographic data, device data, information about the user's response, and includes user input in response to the task provided in response data 114. For example, response data 116 may include the user's selection of a square containing a portion of a bicycle, the amount of time the user spent making the selection, her mouse movement pattern, and her anonymous demographic data, device data, and browsing history, all of which she has allowed system 100 to access. Response data may include semantic descriptors provided by the user regarding the task, product, or design. Semantic descriptors may include any descriptor that provides semantic information about an object such as a product or design. Semantic descriptors may be generated by humans or artificial intelligence and may take the form of words (e.g., keywords or key phrases), sentences, symbols, or other descriptors that convey semantic information. Additionally, semantic descriptors can be assigned to objects based on other actions, such as interacting with presented information (e.g., photos or icons), interacting with rating elements (e.g., product rating tools), or submitting free-form textual feedback about the object. DCDS 110 can then provide this data to data processor 124 for analysis and labeling.

[0078] The process continues to step H, where data processor 124 analyzes response data 116 from user device 106. Data processor 124 is able to analyze response data 116 to classify the data and use user information to label the data, making it searchable. For example, data processor 124 can label the square selected by the user using the user's demographic information, the amount of time she spent making the choice, and the accuracy of her choice (compared to the true set of squares).

[0079] The process continues to step I, in which the data quality processor 120 provides the analyzed data to the tag database 130. The data processor 124 is able to provide tagged data for storage in the tag database 130, making the data searchable.

[0080] System 100 continuously executes process 200, enabling data quality processor 120 to automatically and continuously monitor the quality of data included in a specific dataset. Therefore, system 100 maintains and improves the quality of data in the comprehensive database 130, resulting in a continuous improvement in the accuracy and completeness of model outputs. Because system 100 continuously updates the tag database 130, it provides a searchable database from which system 100 can retrieve content, such as image, audio, or video instances, based on user response information (such as task results) and user information.

[0081] The system reduces bias in user feedback distribution by selectively soliciting additional feedback from underrepresented or unrepresented user demographics.

[0082] Figure 2B and Figure 2C The model training process is described. Server 250 maintains baseline model 252 and task repository 254. Baseline model 252 is the model used as the baseline behavioral model and can be updated. Task repository 254 maintains a set of tasks that can be distributed to users.

[0083] Each of devices A 260a, B 260b, ... and N 260n (collectively referred to as devices 260) includes models A, B, ... N 262a, 262b, ... 262n (collectively referred to as models 262). Each of the locally maintained models 262 can be updated and improved based on tasks and model updates provided by server 250.

[0084] Each device 260n receives input from users 270a, 270b, ... 270n (collectively referred to as users 270), displays information to them, and can be controlled by users 270a, 270b, ... 270n. For example, device 260 can provide each user 270 with the tasks described above. Tasks can be provided from, for example, a task store 254. In response to a task, each user 270 can provide semantic maps 272a, 272b, ... 272n (collectively referred to as semantic maps 272) to device 260. Based on the responses provided by users 270, device 260 can update model 262.

[0085] exist Figure 2C In this process, the model training module 256 can generate an updated baseline model 258 based on the information provided by the response from the user 270, and provide the updated baseline model 258 to the server 250 to replace or update the baseline model 252.

[0086] Figure 3 An example design space 300 is shown. Design space 300 is a visual representation of the conceptual world of possible design values. Design space 300 can be composed of, for example... Figure 1 The system generation is similar to system 100 shown. For example, the model generator 126 of the data quality processor 120 is able to generate design space 300 based on user response data from the tag database 130.

[0087] Design space 300 can be multi-dimensional. In this particular example, design space 300 includes two dimensions and is generated as a result of data submitted by users in response to a question asking them to rate various package designs. In other examples, design space 300 can include more than two dimensions and can be represented as a three-dimensional or multi-dimensional model. Dimensions include design features such as shape, color, texture, size, or relative distance to another object.

[0088] Design space 300 is used to visually represent all possible designs for a specific product or service. Products can include both non-durable and durable consumer goods. For example, products can include clothing, cosmetics, food, furniture, and automobiles.

[0089] Design space 300 can be subject to various constraints, ranging from the physical feasibility of creating the design to its ease of manufacture and designer-imposed constraints. Design space 300 can be limited by data collected from users (such as the product's target audience). For example, if the target consumer group for a particular mobile device would not carry it if it weighed more than three pounds, design space 300 can be limited in its weight dimension. In some implementations, design space 300 can be automatically limited by data and decisions made by system 100. For example, data processor 124 can determine, based on user response data from tag database 130, that none of the surveyed users are interested in laptops with screens smaller than 8 inches, and data processor 124 can provide this limitation on design space 300 to model generator 126.

[0090] By automatically generating and modifying the design space 300, this new system reduces the time and resources required to reach the final design. The system is able to focus design exploration on areas most likely to be productive based on metrics specified by interested third parties. For example, the data quality processor 120 can automatically focus design exploration for a package on the area most likely to be purchased by consumers aged 25 to 34 (the target demographic of the third-party package designer and manufacturer 140). The data quality processor 120 can focus requests for user response data on package designs most interesting to surveyed users indicated to be 25-34 years old by automatically generating package shapes and designs that fall within the design space 300 where users are most likely to be interested.

[0091] In this particular example, design space 300 maps semantic attributes to the geometric features of designs falling within that space. For example, design space 300 maps the semantic attributes “practical” and “fashionable” to specific shapes and forms of bag designs, including bag designs 302 and 304. These semantic attributes are subjective factors representing specific qualitative design goals and demographic information. Based on user response data 306 and 308 from the labeled database 130, data quality processor 120 has determined that users are most interested in bags that combine “practical” and “fashionable” factors. In this particular example, data processor 124 of data quality processor 120 has analyzed user response data from the labeled database 130 and determined that there is significant interest in bags that are more “practical” and at least somewhat “fashionable.” Using this determination, model generator 126 is able to restrict design space 300 to focus on bag designs that exceed a certain threshold amount of “fashionable” and another threshold amount of “practical.” In some implementations, model generator 126 is able to automatically generate designs that meet these design criteria without further input from the designer. In some implementations, model generator 126 is capable of generating package designs that meet previously ungenerated "fashionable" and "practical" thresholds. For example, model generator 126 is capable of generating package designs 312 and 314 without input from third-party package designer 140, and designs 312 and 314 can be new, previously unknown package designs.

[0092] Model generator 126 can generate new designs based on an initial set of input designs by deforming specific geometric features of existing designs to create new designs, for example, using artificial intelligence and machine learning techniques. In some implementations, model generator 126 can generate entirely new designs without using existing designs as a starting point. Data quality processor 120 can be integrated with design programs, including 3D modeling programs and computer-aided design (CAD) programs, to generate new designs or modify existing designs.

[0093] Model generator 126 is capable of using statistical and / or machine learning models that accept user-provided information as input. The machine learning model can use any of a variety of models, such as decision trees, generative adversarial network-based models, deep learning models, linear regression models, logistic regression models, neural networks, classifiers, support vector machines, inductive logic programming, model suites (e.g., using techniques such as bagging, boosting, random forests, etc.), genetic algorithms, Bayesian networks, etc., and can be trained using various methods, such as deep learning, association rules, inductive logic, clustering, maximum entropy classification, learned classification, etc. In some examples, the machine learning model can use supervised learning. In some examples, the machine learning model uses unsupervised learning.

[0094] In some implementations, a probabilistic model can be used, which defines a probabilistic ranking of attributes for a given product design or shape, or a probabilistic ranking of product designs or shapes for one or more attributes.

[0095] Model generator 126 allows designers to explore new areas of the previously unexplored design space and increases the likelihood of a design being well-received by users by automatically generating new designs based on user response data, reducing the amount of time and resource-intensive feedback cycles required for specific products. Designers can then select specific new designs for research and testing. In some implementations, probability ranking can be used to define a set of candidates, which can then be tested in future tasks to refine user preferences. The data collection and design generation process will be described in further detail below. Model generator 126 allows system 100 to characterize user behavioral requirements relative to product design as a function of market demographic information.

[0096] The model generator 126 is capable of using various types of models, including general models applicable to all users and customized models applicable to specific subsets of users sharing a set of features, and can dynamically adjust the model based on received user information of a specific user 102 or based on detected activity. For example, the model generator 126 can use a base station network for a user and then customize a model for each user.

[0097] In some implementations, model generator 126 can use machine learning to determine a target function for a specific user based on subjective feedback from the user. The target function can be simple and implement general modifications to the product design. When system 100 collects user response data, system 100 anonymizes the data and provides it to a central database, which stores and analyzes the collected data to improve the general behavioral model and allows system 100 to provide more personalized strategies for each user 102.

[0098] For example, system 100 can utilize general profiles of users with specific ages, locations, interests, etc. System 100 can generalize support configurations across users predicted to have similar interests. In some implementations, system 100 accepts input from user profile information, such as the user's age, location, and interests, as well as other parameters.

[0099] System 100 can utilize, for example, a personalized "shoe size" model. For instance, System 110 can use general profiles such as people of a specific age group, people in New York, or people who like motorcycles. For example, if multiple users indicate similar preferences, System 100 can determine if a matching product exists. If a matching product exists, System 100 can return the product to the user. If a matching product does not exist, System 100 can modify the existing product design or generate a new design. In some implementations, System 100 can perform aggregated personalization or clustering by mapping specific users to existing user groups or user segments or forming new user groups or user segments. System 100 can also be used to identify product or purchasing trends among specific users. In some implementations, System 100 can perform aggregated configuration clustering by mapping users to existing customer segments or products or mapping user preferences to existing feature sets.

[0100] Furthermore, each model can be personalized. For example, each model can be created from a general model by changing the model parameters based on the characteristics of each user determined from the collected data. Each model can vary over long and short time periods for a specific user. For example, system 100 can track how interested a user is in a particular design element and adjust the behavioral model when it determines that the user has lost interest. In some implementations, each model can also be created from a model that has already been personalized using a general profile and further modified for each user. For example, a model can be created by changing the model parameters based on the characteristics of each user determined from the collected data.

[0101] In some implementations, the model can be personalized without using a base model. For example, user response data can be input into model generator 126 and provided to product designers, manufacturers, or design programs to map to an unadjusted product configuration. In one example, model generator 126 allows a user to immediately purchase a specific item or set an alert when a specific item becomes available.

[0102] Figure 4 It shows Figure 1 Example data flow 400 is shown in the example environment for the design space exploration process. The operation of data flow 400 is performed by various components of system 100. For example, the operation of data flow 400 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tag database 130.

[0103] The process begins in step A, where data processor 124 of data quality processor 120 analyzes and segments the dataset associated with the user group. In some implementations, the dataset is existing data retrieved from a labeled database 130. For example, the dataset could include a group of user responses to a question that requires users to rate two different package designs on a sliding scale relative to two semantic descriptors. User input can be associated with user characteristic data of the user providing the input. For example, user characteristic data could include demographic data and browsing history, as well as other data that system 100 can access and has permissions for.

[0104] The process continues to step B, where data processor 124 identifies missing segments of the dataset based on one or more metrics. Details of this step can be found above relative to step B. Figure 2A It was found in the description.

[0105] The process continues to step C, where model generator 126 generates the design space. As described above... Figure 3 The design space is a visual representation of the total possible designs. In some implementations, the model generator 126 simply updates an existing design space provided by a third party 140. For example, the model generator 126 is able to receive a design space from a third-party package designer 140 and update the design space based on that dataset.

[0106] Model generator 126 is also capable of generating behavioral models that predict user acceptance of a particular design. For example, model generator 126 can utilize artificial intelligence and / or machine learning techniques to generate behavioral models that output evaluations of the design space, as well as portions of the design space that are most likely to be well-received by a particular user group, lack sufficient data points, or are not representative of a particular demographic segment, and other evaluations of a particular metric.

[0107] The process continues to step D, where data processor 124 uses a behavioral model to determine one or more target fragments of the design space. For example, data processor 124 uses the output of the behavioral model generated by model generator 126 in step C to determine whether a very “stylish” but also very “practical” design does not have a threshold number of user reactions to the design, or to determine that no such design has been generated.

[0108] The process continues to step E, where task processor 122 dynamically changes the task to be presented to the user based on one or more determined segments. (See above reference.) Figure 2AThe task processor 122 is capable of modifying existing tasks that have been previously generated and / or presented to the user, or generating entirely new tasks to be presented to the user. For example, the task processor 122 can generate a very "stylish" and very "practical" new package design to present to the user for feedback.

[0109] In some implementations, model generator 126 can display two or more product designs to allow users to visualize the differences or variations between two or more product instances generated by the model.

[0110] In some implementations, the model generator 126 can be integrated with a computer-aided design generation program, and the integrated program can be used to improve, modify, or change the design of a product or service package.

[0111] The process continues to step F, in which task processor 122 transmits dynamically changed or generated tasks to DCDS 110 for distribution to users. Details of this step are available above relative to step D. Figure 2A It was found in the description.

[0112] The process continues to step G, in which DCDS 110 receives a request 108 for content from user equipment 106. Details of this step can be found above relative to step E. Figure 2A It was found in the description.

[0113] The process continues to step H, in which DCDS 110 transmits response data 114 to user equipment 106. Details of this step are available above relative to step F. Figure 2A It was found in the description.

[0114] The process continues to step I, in which DCDS 110 receives response data 116 from user equipment 106. Details of this step are available above relative to step G. Figure 2A It was found in the description.

[0115] The process continues to step J, in which data processor 124 analyzes response data 116 from user equipment 106. Details of this step can be found above relative to step H. Figure 2A It was found in the description.

[0116] The process continues to step K, where model generator 126 updates the design space and / or behavioral model based on the analyzed response data from user device 106. Model generator 126 can narrow or expand the design space based on feedback from the user. For example, model generator 126 can discard a portion of the design space that has been identified as having a threshold number or percentage of user responses and has fewer than the threshold number of positive responses. Model generator 126 can update the behavioral model to reflect the updated dataset. For example, model generator 126 can input the analyzed response data as input to train a behavioral model that predicts user acceptance of a particular package design.

[0117] The data quality processor 120 can also provide analyzed data to the tag database 130. Model and design space updates can be performed concurrently with the data quality processor 120 transmitting the analyzed data to the tag database 130. In some embodiments, these parts of step K can be executed asynchronously.

[0118] System 100 can continuously execute process 400, enabling data quality processor 120 to automatically and intelligently explore the design space. Therefore, system 100 allows for efficient design and development, reducing the time and resources required to generate and complete new designs for products or services.

[0119] The above reference Figure 1 and Figures 3 to 4 The described system automatically modifies tasks to provide data quality improvements. In some embodiments, the data quality processor 120 is capable of modifying the task based on data indicating, for example, specific characteristics of a fragment of data itself, such as a lack of consistent response to one or more types of designs, different image locations, or images with specific characteristics, among other factors. In some embodiments, the data quality processor 120 is capable of modifying the task based on data indicating, for example, specific characteristics of a user group, such as a particular user group spending an unusually short amount of time to complete the task, or a lack of consistent response from the user group, among other factors. System 100 automatically creates product designs based on consumer feedback and obtains further feedback on these designs, allowing for innovative designs that align with consumer preferences and needs without the need for designing and maintaining a focus group. With more representative data, design improvements can be made at a faster pace. Furthermore, additional feedback can be used as input to, for example, training a network of behavioral models to improve the model's classification of what constitutes a positive or negative example.

[0120] Figure 5A and 5B The data flow is described, in which the system uses a behavioral model to generate tasks for the user.

[0121] Figure 5AThe data flow 500 is described, in which the system has user consent to personalize user settings. Figure 1 The tasks received in the example environment. The operation of data stream 500 is performed by various components of system 100. For example, the operation of data stream 400 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tag database 130.

[0122] Process 500 begins with an individual uploading one or more design primitives and / or semantic descriptors, such as keywords or ways of expressing user preferences, such as user clicks or text input (502). The uploaded data can provide information that allows humans to express preferences, semantic descriptions, or ratings, which can be mapped to a given design or more using visual or auditory elements or features of the product or service being designed. For example, market researchers can provide system 100 with a product design for a backpack and a set of keywords associated with the backpack, such as “sports,” “functional,” “practical,” and “professional.” Design primitives can be used for products or services and can include designs for both physical and digital products. In another example, a task can present designs, descriptions, or features to determine the value or price of a product or service. These designs and / or keywords can be stored in a database, such as product designs and keywords. The individual can also upload an experimental plan that determines which design classes or instances should be displayed with which semantic descriptions. The experimental plan can also include statistical measurements or other guidelines for how, when, or where a given design type and semantic description should be displayed. In some implementations, the designs and / or keywords can be stored in a digital component database 112 and / or a tag database 130.

[0123] Process 500 continues, and the system selects the format of the content to be provided to the user (504). For example, the DCDS 110 of system 100 can select the layout of the content to be provided to the user. The layout can include, for example, the types of user interface elements available and the types of information provided. In one example, the DCDS 110 can select a layout for the content that includes sliders and radio buttons for tasks to be provided to the user upon distribution. In another example, the format layout can include a reward to be provided to the user once the user responds.

[0124] The process 500 continues, and the system preprocesses the data and verifies the correctness of the task to be provided to the user (506). For example, the task processor 122 and data processor 124 of the data quality processor 120 can preprocess the data and verify the correctness of the task to be provided to the user based on the layout determined from (504).

[0125] Process 500 continues to store the pre-processed data and verified tasks to be provided to the user (508). In process 500, system 100 has the user's consent to provide personalized content, so the pre-processed data and verified tasks can be modified and / or personalized based on user information.

[0126] Process 500 continues, and the website or application user interacts with the service content (510). For example, DCDS 110 can select preprocessed data and verified tasks to be provided to specific users of the website or application, as described above relative to... Figure 1 As described, and receives user input from the user's interaction with the content of the service.

[0127] Process 500 continues, storing the user's response (512). For example, DCDS 110 can receive a user's response, which includes user information such as user demographics after the response data has been analyzed and processed by data processor 124 of data quality processor 120. The analyzed response data can be tagged by data processor 124 and stored in tag database 130.

[0128] Process 500 continues, constructing one or more behavioral models (514) based on user response data. Examples of behavioral models include using linear or nonlinear function approximations (e.g., deep learning-enabled convolutional neural networks) to map designs, design features, design variations, or graph-based design representations to semantic descriptions or ratings. Reversible models that allow mappings from input to output and vice versa can be used, or multiple models that can map inputs to outputs and vice versa can be used. Behavioral models may also include user demographic information stored along with the responses of a given user. For example, a model could be created using task data that allows viewing clothing design feedback from women aged 20 to 30 living in London. For example, a behavioral model could use user demographic information as input, or be inherently probabilistic, allowing the computer to tune the model response for a given set of demographic parameters. For example, the model generator 126 of the data quality processor 120 is capable of generating behavioral models based on user response data, as described above relative to... Figures 1 to 4 The model-based or model-driven analytics may also include information or input from other sources, such as online surveys, pricing and conversion data, sales attribution models, topic clinics, and focus group feedback. Model-based or model-driven analytics may also utilize pricing information obtained from task feedback.

[0129] Process 500 continues, analyzing and identifying new ideas and / or concepts based on one or more behavioral models (516). For example, the model generator 126 of the data quality processor 120 is able to analyze and identify new design ideas and / or new design concepts based on the output of the behavioral models.

[0130] Process 500 continues, exploring and / or optimizing the design to generate new designs and / or keywords or semantic descriptors or preferences (518). For example, the model generator 126 of the data quality processor 120 can use the design space to explore and / or optimize the design to generate new designs or modify existing designs. The model generator 126 can also generate new keywords associated with the design. For example, the model generator 126 can generate new keywords that are trending or popular in current marketing materials and have been shown to be popular with users, based on the generated behavioral model. The model can be used to generate designs from semantic descriptions or ratings, and can be used to generate semantic descriptions or ratings from a given design. The model can be used to explore or optimize the model space to identify new designs or semantic descriptions and include them in future tasks. Figure 3 An example can be seen where the model space is shown as a design positioned relative to two axes associated with keywords (fashion and practicality). In this example, design 312 can be selected to optimize a cost function where practicality accounts for 60% and fashion accounts for 40%. An example includes the use of a behavioral model that includes pricing feedback, from which optimization can be safely obtained from a task that considers design trade-offs, costs, and pricing. These new designs and / or keywords are provided to a database of product designs and keywords referenced in (502). In some implementations, the designs and / or keywords can be stored in a digital component database 112 and / or a tag database 130.

[0131] Figure 5B Data flow 550 is described, in which the system personalizes user data without user consent. Figure 1 The task received in the example environment. The operation of data stream 550 is performed by various components of system 100. For example, the operation of data stream 550 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tag database 130.

[0132] Process 550 follows immediately after process 500. Process 550 begins with the client uploading one or more design primitives and / or keywords (552). Details of this step can be found above relative to (502). Figure 5A It was found in the description.

[0133] Process 550 continues, and the system selects the format of the content to be provided to the user (554). Details of this step can be found above relative to (504). Figure 5A It was found in the description.

[0134] Process 550 continues, the system preprocesses the data and verifies the correctness of the task to be provided to the user (556). Details of this step can be found above relative to (506). Figure 5A It was found in the description.

[0135] Process 550 continues, storing the pre-processed data and verified tasks to be provided to the user (558). In process 550, system 100 does not have the user's consent to provide personalized content, therefore neither the pre-processed data nor the verified tasks are modified or personalized based on user information.

[0136] Process 550 continues, with the website or application user interacting with the service content (560). Details of this step can be found above relative to (510). Figure 5A It was found in the description.

[0137] Process 550 continues, storing the user's response (562). For example, DCDS 110 can receive the user's response, which may include user information, such as user demographics after the response data has been analyzed and processed by data processor 124 of data quality processor 120. If user information is included, DCDS 110 can remove the user information. In some implementations, user information is not provided because system 100 does not have the user's consent to access the user information. The analyzed response data can be tagged by data processor 124 and stored in tag database 130.

[0138] Process 550 continues, constructing one or more behavioral models (564) based on user response data. Details of this step can be found above relative to (514). Figure 5A It was found in the description.

[0139] Process 550 continues, analyzing and identifying new ideas and / or concepts based on one or more behavioral models (566). Details of this step can be found above relative to (516). Figure 5A It was found in the description.

[0140] Process 550 continues, exploring and / or optimizing the design to generate new design and / or semantic descriptors (568). Details of this step can be found above relative to (518). Figure 5A It was found in the description.

[0141] Figure 6A and Figure 6B It describes the data flow in which the system integrates user feedback into the design cycle. Figure 6A and Figure 6B Based on Figure 5A and Figure 5BThe data stream shown. In some embodiments, system 100 has user consent for personalized content. In some embodiments, system 100 does not have user consent for personalized content.

[0142] Figure 6A The data flow 600 is described, in which the system integrates user feedback into the design cycle, conforming to... Figure 1 The example environment provides designer input. The operation of data flow 600 is performed by various components of system 100. For example, the operation of data flow 600 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tag database 130.

[0143] Process 600 follows either process 500 or 550. Process 600 begins with the individual selecting the task format and uploading the design primitives and semantic descriptors (602). Details of this step can be found above relative to (502). Figure 5A Or relative to (552) Figure 5B The description can be found there. For example, users can upload design primitives or design shapes along with associated keywords to the database. The database stores the initial product design shape and keywords, as well as any updates based on the user's response to the viewing task. For example, the database can store updates based on step 618.

[0144] Process 600 continues, and the system selects the format of the content to be provided to the user (604). Details of this step can be found above relative to (504). Figure 5A Or relative to (554) Figure 5B The description can be found therein. The format of the primitives and semantic descriptors stored is chosen by a person or can be automatically configured by the system.

[0145] Process 600 continues, the system preprocesses the data and verifies the correctness of the task to be provided to the user (606). Details of this step can be found above relative to (506). Figure 5A Or relative to (556) Figure 5B The data can be preprocessed, for example, to identify a set of keywords available for a specific design shape. Furthermore, the appropriateness or correctness of the data can be validated for a given format or the user's intended audience. In some implementations, the task may be relevant to specific country, geographic, or demographic characteristics. For example, interpreting a French language description might not be a relevant task for non-French language users in much of the United States and Europe. These tasks can be stored and delivered along with content distributed via the Internet or applications.

[0146] Process 600 continues, storing the preprocessed data and validated task to be provided to the user (608). Details of this step can be found above relative to (508). Figure 5A Or relative to (558) Figure 5B It was found in the description.

[0147] Process 600 continues, where the user of the website or application interacts with the content of the service (610). Details of this step can be found above relative to (510). Figure 5A Or relative to (560) Figure 5B The description can be found there. For example, a user interacts with the content and issues a task request. The user can then interact with the task and provide a response. For example, a user can select the most suitable keywords or tags for a photo or image of a product design.

[0148] Process 600 continues, storing the user's response (612). Details of this step can be found above relative to (512). Figure 5A Or relative to (562) Figure 5B The description can be found therein. In some implementations, only the response is logged. In some implementations, both the response and user demographics may be stored.

[0149] Process 600 continues, constructing one or more behavioral models (614) based on user response data. Details of this step can be found above relative to (514). Figure 5A Or relative to (564) Figure 5B The description can be found within the model. For example, a model can map keywords and semantic descriptors to specific design shapes or geometries. In another example, a model maps product features to preferred sets of other features, or products to preferred sets of other products. These models can also map design features or decisions to customer preferences and values.

[0150] Process 600 continues to analyze and identify new ideas and / or concepts based on one or more behavioral models (616). Details of this step can be found above relative to (516). Figure 5A Or relative to (566) Figure 5B The description can be found there. For example, analyzing and identifying models to generate data and results that can be used to help determine new tasks.

[0151] Process 600 continues, exploring and / or refining the design to generate new designs and / or keywords (618). Details of this step can be found above relative to (518). Figure 5A Or relative to (568) Figure 5BThe description can be found therein. For example, in response to this analysis, the system can use the data to explore and optimize or modify task content, format, or messaging. Any learning, modifications, and updates related to models, analyses, design decisions, and optimizations are stored in the database.

[0152] Process 600 includes loading a behavioral model into a CAD-based design tool and making product design decisions based on the behavioral model (620). In some implementations, human designers can provide input to the CAD-based design tool and / or provide product design decisions. For example, human designers can implicitly have a desired semantic description of the outcome in mind and can form the output of the CAD-based design tool and automatically generated product design decisions by the model generator 126. The CAD-based design tool can also receive constraints and specifications provided by, for example, human designers, product manufacturers, or customers. In some implementations, models that capture user feedback or behavior can be used directly or indirectly for computer-aided design decisions and manufacturing production. These models can be used to analyze or modify the design.

[0153] Process 600 continues, generating a modified design shape (622). For example, model generator 126 can use a behavioral model and the product's design space to generate a modified design shape. In some implementations, the model can map keyword descriptions to actual 3D product designs, and designers can use keyword descriptions and user feedback to generate new or modified designs. The design can be represented as an explicit shape, a data structure representing a shape or its deformation, or a shape diagram. In one example, designers use the model to create or modify designs based on other products, which may be sold or bundled. This cycle can be repeated until product requirements or release guidelines are met. If they are met, the design is provided as specifications, manufacturing directions, recipes, or a method to guide or direct the manufacturing process. The resulting product can be sent to the original user who provided feedback or provided to a warehouse or store for distribution to customers. In some implementations, the method can be provided manually, where information is provided to the person making the decision. In some implementations, the method can be used to automatically modify the design based on constraints, specifications, or a cost function that guides the design toward a better state. In some implementations, the method can use a hybrid approach, with some steps performed manually and others automatically.

[0154] The process 600 continues to determine whether the modified design shape meets the release criteria (624). For example, the model generator 126 can determine whether the design shape is suitable for release or manufacturing based on release criteria from, for example, product designers, manufacturers, shippers, etc.

[0155] Process 600 may optionally include manufacturing and shipping products with a modified design shape (626). For example, the design process may be integrated with the product manufacturing and distribution workflow. In some implementations, products may be distributed to warehouses, stores, and directly to users with similar preferences to the user who provided the initial response, such as other users in the same group, cluster, location, or user group in other shared groups.

[0156] Figure 6B The data flow 650 is described, in which the system integrates user feedback into the design cycle to... Figure 1 New designs are automatically generated in the example environment. The operation of data flow 650 is performed by various components of system 100. For example, the operation of data flow 650 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tag database 130.

[0157] A key feature of process 650 is that generative design can be used to enhance the vision of human designers and, in some implementations, can completely replace human designers based on desired results.

[0158] Process 650 follows immediately after processes 500, 550, or 600. Process 650 begins with the client uploading one or more design primitives and / or keywords (652). Details of this step can be found above relative to (502). Figure 5A Compared to (552) Figure 5B Or relative to (602) Figure 6A It was found in the description.

[0159] Process 650 continues, and the system selects the format of the content to be provided to the user (654). Details of this step can be found above relative to (504). Figure 5A Compared to (554) Figure 5B Or relative to (604) Figure 6A It was found in the description.

[0160] Process 650 continues, the system preprocesses the data and verifies the correctness of the task to be provided to the user (656). Details of this step can be found above relative to (506). Figure 5A Compared to (556) Figure 5B Or relative to (606) Figure 6A It was found in the description.

[0161] Process 650 continues, storing the preprocessed data and validated task to be provided to the user (658). Details of this step can be found above relative to (508). Figure 5A Compared to (558) Figure 5BOr relative to (608) Figure 6A It was found in the description.

[0162] Process 650 continues, with the website or application user interacting with the service content (660). Details of this step can be found above relative to (510). Figure 5A Compared to (560) Figure 5B Or relative to (610) Figure 6A It was found in the description.

[0163] Process 650 continues, storing the user's response (662). Details of this step can be found above relative to (512). Figure 5A Compared to (562) Figure 5B Or relative to (612) Figure 6A It was found in the description.

[0164] Process 650 continues, constructing one or more behavioral models based on user response data (664). Details of this step can be found above relative to (514). Figure 5A Compared to (564) Figure 5B Or relative to (614) Figure 6A It was found in the description.

[0165] Process 650 continues, analyzing and identifying new ideas and / or concepts based on one or more behavioral models (666). Details of this step can be found above relative to (516). Figure 5A Compared to (566) Figure 5B Or relative to (616) Figure 6A It was found in the description.

[0166] Process 650 continues, exploring and / or optimizing the design to generate new design and / or semantic descriptors (668). Details of this step can be found above relative to (518). Figure 5A Compared to (568) Figure 5B Or relative to (618) Figure 6A It was found in the description.

[0167] Process 650 includes loading the behavioral model into a CAD-based design tool and making product design decisions based on the behavioral model (670). In some implementations, generative design processes can be used to automatically generate or modify designs based on desired criteria or cost functions. Such generative design processes can provide input to CAD-based design tools and / or provide product design decisions. For example, a generative design system can determine or receive an appropriate cost function based on desired results. CAD-based design tools can also receive constraints and specifications provided by, for example, human designers, product manufacturers, or customers. For example, artificial intelligence methods can be used in conjunction with iterative generative design to search the design space and generate designs that meet specifications and behavioral model results.

[0168] Process 650 continues, producing the modified design shape (672). Details of this step can be found above relative to (622). Figure 6A It was found in the description.

[0169] Process 650 continues, determining whether the modified design shape meets the release criteria (674). Details of this step can be found above relative to (624). Figure 6A It was found in the description.

[0170] Process 650 may optionally include manufacturing and shipping the product with the modified design shape (676). Details of this step are available above relative to (626). Figure 6A It was found in the description.

[0171] In one example, the task could involve showing a user two different products, such as shoes, and asking the user which pairs of shoes best match the displayed outfit. For example, the outfit could be a set. The outfit could include multiple different pieces and ask the user to select a set.

[0172] In one example, a task could include a way for users to configure or modify the design and solicit feedback on how to create a new product or improve an existing one. The improved product could then be offered to users in a new task, allowing them to iteratively refine the product design.

[0173] In one example, the task could include showing the user two different product designs, such as a car design, and providing the user with feedback mechanisms, such as sliders or buttons, to provide a rating of the user's level of trust in a given attribute, such as a subjective product design descriptor (or adjective) like "compact" describing each design.

[0174] Generative design can automatically drive a computer-aided design process to generate and modify designs based on criteria and specifications. Cost functions can be used to weigh design criteria or factors within constraints and specifications. Behavioral models can be used in conjunction with cost functions to maximize user preferences within constraints, specifications, or manufacturing rules. For example, generative design methods can be used to iterate across the design space guided by behavioral models and cost functions or decision criteria. The use of generative design involves using AI and reinforcement learning to iterate across numerous design decisions that are permissible or within specifications or constraints but may represent different customer preferences, such as the body style of a car or the color combination of running shoes. Generative algorithmic methods include one or more evolutionary algorithms, variational autoencoders, and generative adversarial networks. These methods can leverage cloud computing to iterate a large number of design iterations to optimize a single customer, a group of customers, or multiple customer groups. Examples of algorithms that can be used include evolutionary algorithms, which include genetic algorithms that evolve a given design or create a mixture of multiple designs. In some implementations, the system can use generative adversarial networks to generate entirely new designs from inputs of multiple existing designs. These methods can iteratively generate new designs, with the task of presenting the new designs to the user to provide relevant feedback. The entire process can be automatically iterated and optimized to generate new designs. In some implementations, the system can be integrated into automated manufacturing processes using robots or 3D printing. In other implementations, the system can be used to personalize products for users or user groups based on preferences provided by them.

[0175] Variational autoencoders and associated generative adversarial networks can be used with task-generated behavioral models to develop new designs, which are then fed into the system to obtain additional user feedback. The system iterates and is scored until a stopping criterion defined by a cost function is met.

[0176] Variational autoencoders (VAEs) can be used with design libraries that represent images, shapes, shape-related data structures, shape-related graphics, or polygonal data. A VAE can be used to create a set of latent factors that effectively represent a reduced set of features describing a given design. Once trained, the design is encoded into its latent factors and then decoded into its reconstructed design. In one example, an existing design representation library can be used to build an initial mapping to accelerate convergence. Behavioral models developed using task-based feedback can be used to map a given reconstructed design representation to user features or classifications. In some implementations, user classification can be a good (preferred) to poor (non-preferred) rating. User / human-based representations can be used with a cost function to create mathematical representations or scores that an optimizer can use to create or improve designs. The optimizer's output is the design type associated with the latent factor description. Optimization can even use mutations or crossovers to create new latent factors. New latent factors can be used by a decoder / generator to produce new design representations that can be inserted into new tasks. Design, latent factors, and keywords all form a space for design and distance metrics. When creating new tasks to present to users, this space can be used to associate or cluster one design with another. The system can automatically iterate around the design space until the cost-based scoring converges to very small values, where the resulting latent factors meet stopping criteria or require modification. At this point, the design is considered complete and ready for production.

[0177] In another implementation, creating or optimizing a design can include creating a mathematical representation of a design space, where each car or clothing design is represented by design features defined by each dimension of the space. The N-dimensional feature space is represented in an N-space. In many cases, this space can be transformed into a lower-dimensional space. In other cases, distance metrics can be used to cluster or segment design instances into classes. The distance from each design instance or class to another instance or class can be calculated and stored. The behavioral model represents a mapping between design instances and a set of keywords or semantic descriptions for the task evaluator / user. This mapping can be explored to find designs similar to the most promising designs evaluated so far (optimization) or designs dissimilar to those (exploration), thereby discovering and exploring new parts of the design space. The system can be used with distance-based clustering methods, such as K-nearest neighbors or collaborative filtering, to define design instances for future tasks displayed to the user.

[0178] Figure 7A and Figure 7B It describes the data flow in which the system implements user feedback to customize existing designs and products.

[0179] Figure 7A Data flow 700 is described, in which the system is... Figure 1User feedback is implemented in the example environment to modify existing designs and products. The operation of data flow 700 is performed by various components of system 100. For example, the operation of data flow 700 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tag database 130.

[0180] Dataflow 700 can be integrated with various manufacturing processes, including 3D printing or automated manufacturing.

[0181] Process 700 follows immediately after processes 500, 550, 600, or 650. Process 700 begins with the client uploading one or more design primitives and / or keywords (702). Details of this step can be found above. Figure 5A Compared to (502), Figure 5B Compared to (552), Figure 6A Compared to (602) or Figure 6B It was found in the description relative to (652).

[0182] Process 700 continues, and the system selects the format of the content to be provided to the user (704). Details of this step can be found above. Figure 5A Compared to (504), Figure 5B Compared to (554), Figure 6A Compared to (604) or Figure 6B It was found in the description relative to (654).

[0183] Process 700 continues, the system preprocesses the data and verifies the correctness of the task to be provided to the user (706). Details of this step can be found above. Figure 5A Compared to (506), Figure 5B Compared to (556), Figure 6A Compared to (606) or Figure 6B It was found in the description relative to (656).

[0184] Process 700 continues, storing the preprocessed data and validated task to be provided to the user (708). Details of this step can be found above. Figure 5A Compared to (508), Figure 5B Compared to (558), Figure 6A Compared to (608) or Figure 6B It was found in the description relative to (658).

[0185] Process 700 continues, where the website or application user interacts with the service content (710). Details of this step can be found above. Figure 5A Compared to (510), Figure 5B Compared to (560), Figure 6A Regarding (610) or Figure 6BIt was found in the description relative to (660).

[0186] Process 700 continues, storing the user's response (712). Details of this step can be found above. Figure 5A Compared to (512), Figure 5B Compared to (562), Figure 6A Compared to (612) or Figure 6B It was found in the description relative to (662).

[0187] Process 700 continues, building one or more behavioral models based on user response data (714). Details of this step can be found above. Figure 5A Compared to (514), Figure 5B Compared to (564), Figure 6A Compared to (614) or Figure 6B It was found in the description relative to (664).

[0188] Process 700 continues, analyzing and identifying new ideas and / or concepts based on one or more behavioral models (716). Details of this step can be found above. Figure 5A Compared to (516), Figure 5B Compared to (566), Figure 6A Compared to (616) or Figure 6B It was found in the description relative to (666).

[0189] Process 700 continues, exploring and / or refining the design to generate new design and / or semantic descriptors (718). Details of this step can be found above. Figure 5A Compared to (518), Figure 5B Compared to (568), Figure 6A Compared to (618) or Figure 6B It was found in the description relative to (668).

[0190] Process 700 includes using a behavioral model to identify existing products or product suites that have features most closely resembling the design identified by the behavioral model (720). In some implementations, the model generator 126 has access to a database or other searchable structure that stores existing product designs and features with or without pre-existing keyword descriptions.

[0191] Process 700 continues, generating the modified design shape (722). Details of this step can be found above. Figure 6A Relative to (622) or Figure 6B It was found in the description relative to (672).

[0192] Process 700 continues, determining whether the modified design shape meets the release guidelines (724). Details of this step can be found above. Figure 6A Compared to (624) or Figure 6B It was found in the description relative to (674).

[0193] Process 700 may optionally include manufacturing and shipping products with the modified design shape (726). Details of this step are available above. Figure 6A Compared to (626) or Figure 6B It was found in the description relative to (676).

[0194] In one example, the task involves displaying a set of product designs. For instance, the set could include different product designs for different project types, such as interior furniture (e.g., designs for chairs, tables, beds, accessories, artwork, etc.), and the user would be asked to select which of the different product designs and project types they found visually appealing together. In some implementations, user feedback can be used in subsequent tasks to test existing and develop new marketing strategies, such as cross-selling and upselling strategies.

[0195] In another example, the task includes providing a way for users to design or configure products (such as shoes or cars), and a way for users to receive products or for manufacturers to manufacture products and ship them to users. For example, users can provide input through feedback mechanisms to indicate new designs or designs configured using a predetermined set of characteristic values.

[0196] In another example, the task involves users modifying existing product designs and having manufacturers provide users with ways to customize products (e.g., using a 3D printer). For instance, users could provide input through a feedback mechanism to indicate modifications to the existing design.

[0197] Figure 7B Data flow 750 is described, in which the system implements user feedback to... Figure 1 Custom software design and products are provided in the example environment. The operation of data flow 750 is performed by various components of system 100. For example, the operation of data flow 750 can be performed by data quality processor 120, which communicates with DCDS 110, user equipment 106, third party 140, and tag database 130.

[0198] Dataflow 750 uses a semantic behavior model to configure software products to suit, for example, specific users, user groups, or communities. In one example, Dataflow 750 is used to create mini-games with characters, weapons, environments, and / or situations, as well as other features, selected or customized based on user information.

[0199] Process 750 follows processes 500, 550, 600, 650, or 700. Process 750 begins with the customer uploading one or more design primitives and / or semantic descriptors (752). Details of this step can be found above. Figure 5A Compared to (502), Figure 5B Compared to (552), Figure 6A Compared to (602), Figure 6B Compared to (652) or Figure 7A It was found in the description relative to (702).

[0200] Process 750 continues, and the system selects the format of the content to be provided to the user (754). Details of this step can be found above. Figure 5A Compared to (504), Figure 5B Compared to (554), Figure 6A Compared to (604), Figure 6B Compared to (654) or Figure 7A It was found in the description relative to (704).

[0201] Process 750 continues, the system preprocesses the data and verifies the correctness of the task to be provided to the user (756). Details of this step can be found above. Figure 5A Compared to (506), Figure 5B Compared to (556), Figure 6A Compared to (606), Figure 6B Compared to (656) or Figure 7A It was found in the description relative to (706).

[0202] Process 750 continues, storing the preprocessed data and validated task to be provided to the user (758). Details of this step can be found above. Figure 5A Compared to (508), Figure 5B Compared to (558), Figure 6A Compared to (608), Figure 6B Compared to (658) or Figure 7A It was found in the description relative to (708).

[0203] Process 750 continues, with the website or application user interacting with the service's content (760). Details of this step can be found above. Figure 5A Compared to (510), Figure 5B Compared to (560), Figure 6A Compared to (610), Figure 6B Compared to (660) or Figure 7A It was found in the description relative to (710).

[0204] Process 750 continues, storing the user's response (762). Details of this step can be found above. Figure 5A Compared to (512), Figure 5B Compared to (562), Figure 6A Compared to (612), Figure 6B Compared to (662) or Figure 7A It was found in the description relative to (712).

[0205] Process 750 continues, constructing one or more behavioral models based on user response data (764). Details of this step can be found above. Figure 5A Compared to (514), Figure 5B Compared to (564), Figure 6A Compared to (614), Figure 6B Compared to (664) or Figure 7A It was found in the description relative to (714).

[0206] Process 750 continues, analyzing and identifying new ideas and / or concepts based on one or more behavioral models (766). Details of this step can be found above. Figure 5A Compared to (516), Figure 5B Compared to (566), Figure 6A Compared to (616), Figure 6B Compared to (666) or Figure 7A It was found in the description relative to (716).

[0207] Process 750 continues, exploring and / or refining the design to generate new design and / or semantic descriptors (768). Details of this step can be found above. Figure 5A Compared to (518), Figure 5B Compared to (568), Figure 6A Compared to (618), Figure 6B Compared to (668) or Figure 7A It was found in the description relative to (718).

[0208] Process 750 includes using behavioral models to configure software or digital products for personalization or customization based on user preferences (770). In some implementations, a configurator is used to personalize the software product. In some implementations, the software product can be a mini-game. In some implementations, the software product can be embedded in a web page or application.

[0209] For example, if user information indicates that a particular user likes the Star Wars franchise and the software product is a mini-game, then the data quality processor 120, and in particular the model generator 126, can modify the mini-game to include Star Wars characters, audio, items, etc., with the permission of the user and Lucasfilm.

[0210] In another example, game developers take an existing or new game that may have many levels, characters, and weapons, and allow users to select parts that can be configured using user feedback through missions, allowing a small portion to be played online or downloadably.

[0211] Step 750 continues, determining whether the revised game design meets the release guidelines (772). Details of this step can be found above. Figure 6A Compared to (624), Figure 6B Compared to (674) or Figure 7A It was found in the description relative to (724).

[0212] Process 750 may optionally include distributing or providing download options with the modified game design to users interacting with the system, providing feedback (774) to other users, and the actual delivery of the modified game design (775). Details of this step can be found above. Figure 6A Compared to (626), Figure 6B Compared to (676) or Figure 7A It was found in the description relative to (726).

[0213] In one example, the task involves showing a user two different software game characters or attributes related to a game application, such as window size, and asking the user which one they are more interested in. In some examples, the method can also include offering the user software pre-configured with the selected character or attribute. In another example, the method can include offering users of online games software pre-configured with the selected character or attribute. In yet another example, behavioral models are created using task feedback from one or more users to design new software such as applications or games, which can take various forms with characters, settings, and backgrounds tailored to the demographics of one or more user groups. In yet another example, multiple users can collaboratively design game features to be inserted into a multiplayer gaming environment accessible to the users using tasks.

[0214] Process 750 may optionally include distributing or providing download options with the modified game design to users other than those interacting with the system, and providing feedback (776) and actual delivery of the modified game design to other users (777). Implementation details are the same as (774).

[0215] Figure 7C Here is a concrete example where the system incorporates user feedback into the design cycle, such as relative to... Figure 7BAs described. The system is capable of receiving products such as software titles (780). For example, the system is capable of receiving games or applications. The system is capable of identifying product attributes and assets. For example, the system is capable of identifying game attributes, including characters, weapons, scenes, environments, keywords, etc. The system is capable of identifying game assets, including thumbnails, demo versions, multiple configurations, usage / game videos, reviews, descriptions, keywords, super-random versions, etc. The system is capable of presenting game attributes to users (782). For example, the system is capable of presenting users with character A, character B, weapon A, and weapon B. Users are able to make their choices and choose to play the game. The system is capable of presenting game assets to users (783). For example, the system is capable of presenting users with thumbnails. Figure 1 The screen includes a thumbnail and two sliders that prompt the user for input. The first slider asks, "Which game does this label represent: 'Role-playing'?" The user can drag the slider to the side indicating their answer. The second slider asks, "Which game does this label represent: 'Jigsaw Puzzle'?"

[0216] Figure 7D This is a concrete example where the system incorporates user feedback into the design cycle, such as relative to... Figure 7B As described. The user can interact with the task to select settings or configuration preferences (784). Task-related UI elements can be presented to the user (785). The game or application can be configured using the processes outlined in (786)-(788). For example, the process can include settings, configurations, or game state descriptions such as character / skin, inventory / load, settings / environment, and tasks (786). These settings and configurations can be encrypted to control usage and restrict permissions to sub-games. The engine can then take input and use the settings, configurations, or game state descriptions to generate a usable and workable environment (787). The game or application can then be a configured / personalized experience (788). The configured game or application can be provided to the user via task-related UI elements (789). The user can then receive or download the configured game or application (790). The user can be the same user interacting with the task in (784) or a different user.

[0217] Figure 7E This is a concrete example where the system incorporates user feedback into the design cycle, such as relative to... Figure 7BAs described. Users can interact with tasks to select settings or configuration preferences (791). Task-related UI elements can be presented to the user (792). Games or applications can be configured using the processes outlined in (793)-(795). For example, the process can include settings, configurations, or game state descriptions such as character / skin, inventory / load, settings / environment, tasks (793). These settings and configurations can be encrypted to control usage and restrict permissions to subgames. The engine can then map user settings, configurations, or game state preferences to pre-configured or pre-compiled games / applications from the library (794). The game or application can then be a configured / personalized experience (795). The configured game or application can be provided to the user via task-related UI elements (796). The user can then receive or download the configured game or application (797). The user can be the same user interacting with the task in (791) or a different user.

[0218] Figure 7F This is a specific example of a system using an autoencoder to incorporate user feedback into design feedback. As described above, the autoencoder can be used with a design library to create a set of latent factors that effectively represent a reduced set of features describing a given design. The system has access to a library of design representations, including shapes, images, charts, data structures, etc. (A). The system can then use the set of latent factors that effectively represent a set of features describing the design to generate a reconstructed design (B). The system can automatically create car designs from raw data or from evolved latent factors or improved design features (C). The system can receive results from a behavioral model created through a user-defined task (D). The system can use the behavioral model along with a semantic representation of the car design for user classification (e.g., sporty, family, compact) (E). The system uses a cost function-based scoring method (F) and performs optimization to create new feature vectors to automatically create design objects (G). The method terminates if the optimization converges to a specific set of criteria (H).

[0219] Figure 8 This is a flowchart of an example process 800 for data quality improvement. In some implementations, process 800 can be executed by one or more systems. For example, process 800 can be performed by... Figure 1 To Figure 2 and Figure 4 The process 800 may be implemented by a data quality processor 120, a DCDS 110, a user equipment 106, and a third party 140. In some embodiments, the process 800 may be implemented as instructions stored on a computer-readable medium, which may be non-transitory, and when executed by one or more servers, the instructions may cause one or more servers to perform the operations of the process 800.

[0220] Process 800 begins by receiving a request (802) from the user equipment for a digital component to be presented at the user equipment. For example, system 100 may receive request 108 from user equipment 106.

[0221] Process 800 continues, determining one or more attributes of the user based on one or more pieces of information provided by the user or information included in a request for digital components (804). As described above relative to... Figure 2A The data processor 124 discussed herein can determine one or more attributes of a user based on information provided by the user, such as a user profile or information contained in a request. For example, information provided in a user profile may include characteristics such as the user's age, gender, interests, and location. Information contained in request 108 may include anonymous information or information that cannot be used to identify the user, and may include, for example, the website that generated request 108 and the destination website the user navigated to. In one example, data processor 124 can determine that the user is a male older than 65.

[0222] Process 800 continues by identifying behavioral models corresponding to one or more attributes of the user (806). For example, model generator 126 is able to identify behavioral models that predict user behavior based on one or more attributes of the user. In one example, model generator 126 is able to identify behavioral models for men aged 65 and over.

[0223] Process 800 continues, dynamically altering the presentation of the item depicted by the digital components based on an identified behavioral model corresponding to one or more attributes of the user (808). For example, task processor 122 and / or data processor 124 are capable of dynamically modifying the presentation of the item depicted by the digital components, such as a task question, based on an identified behavioral model corresponding to one or more attributes of the user. In one example, task processor 122 is capable of modifying the task question about a mug design depicted in the task based on a behavioral model of a male older than 65.

[0224] In some implementations, task processor 122 selects the format of the feedback solicitation based on the digital components of the feedback solicited from the user regarding the project. For example, task processor 122 selects the format of the user's reaction to a particular mug design.

[0225] In some implementations, the data quality processor 120 dynamically alters the presentation of an item depicted by digital components based on identified behavioral models corresponding to one or more attributes of a user, including using machine learning or artificial intelligence techniques to identify feedback to be solicited from the user regarding the item. For example, the data quality processor 120 can use the output of the model generator 126 to identify feedback to be solicited from the user regarding the mug design.

[0226] In some implementations, task processor 122 verifies the information solicited by the digital component based on specific attributes corresponding to underrepresented portions of the user population. For example, task processor 122 verifies that the information solicited by the task will be distributed based on the user's age attribute.

[0227] Process 800 continues, determining that the user has a specific attribute corresponding to an underrepresented portion of the user group in the database containing information about the item (810). For example, data processor 124 can determine that the portion of male users older than 65 years old is an underrepresented portion based on the number of threshold responses. In some embodiments, data processor 124 uses statistical analysis to identify the underrepresented portion of the user group. Data processor 124 is then able to determine that the user of user device 106 has an age attribute corresponding to an underrepresented portion of the user group in the database containing information about the mug design.

[0228] Process 800 continues, in response to determining that a user has a specific attribute corresponding to an underrepresented portion of the user group, and in response to that request, generating a digital component that includes a presentation of dynamic changes to the project, soliciting feedback from the user regarding the project, and including a feedback mechanism (812) that enables the user to submit feedback on the project. For example, task processor 122 is capable of generating a presentation of changes to the project or a changed task that solicits feedback from the user device 106 regarding the mug design, and includes a feedback mechanism, such as a voting feature, that enables the user to submit feedback on the project. In some implementations, DCDS 110 generates or selects digital components to be distributed to user device 106, as described above relative to... Figure 1 As shown in Figure 7.

[0229] Process 800 continues, updating the database to include feedback from the user regarding the project (814). For example, data processor 124 is able to use user feedback from user device 106 to update the database containing information about the project.

[0230] In some implementations, in response to receiving feedback from a user having specific attributes that correspond to an underrepresented portion of the user group, the data quality processor 120 uses one or more of the user's attributes to tag the feedback information and stores the tagged feedback information in a tagged, searchable database, such as a tag database 130.

[0231] Process 800 continues, at least in part based on feedback obtained from the user, modifying the presentation of the item when it is distributed to other users who share one or more attributes of that user (816). For example, data quality processor 120 is capable of modifying the presentation of the item when it is distributed to other users who share one or more attributes of the user, based on feedback obtained from the user. This allows data quality processor 120 to adjust its tasks and product design based on the attributes of the users interacting with the system.

[0232] In some implementations, the data quality processor 120 is able to modify the presentation of an item when it is distributed to other users with one or more attributes of the user by selecting specific feedback mechanisms included in the digital components.

[0233] Figure 9 This is a flowchart of an example process 900 for automatically designing space exploration. In some implementations, process 900 can be executed by one or more systems. For example, process 900 can be performed by... Figure 1 To Figure 2 and Figure 4 The process 900 is implemented by a data quality processor 120, a DCDS 110, a user equipment 106, and a third party 140. In some embodiments, the process 900 can be implemented as instructions stored on a computer-readable medium, which may be non-transitory, and when executed by one or more servers, the instructions enable one or more servers to perform the operations of the process 900.

[0234] Process 900 begins by receiving a request (902) from the user equipment for a digital component to be presented at the user equipment. For example, as described above, system 100, particularly DCDS 110, is capable of receiving a request 108 for a digital component to be presented at user equipment 106.

[0235] Process 900 continues, receiving a dataset (904) of user-provided information regarding a specific product design. For example, data quality processor 120 can receive user-provided responses regarding a specific product design, such as a handbag design. In some implementations, the product design is specific to a particular product. The product design can be a service or a software product. For example, a specific product design can be the user interface design of a software application.

[0236] Process 900 continues, generating a visual representation (906) based on the dataset of information provided by the user, mapping design factors to continuous shapes representing the geometry of the potential product design. As above, relative to... Figures 3 to 4 The model generator 126 is capable of generating a design space that maps subjective factors to continuous shapes representing potential product designs. For example, the model generator 126 is capable of generating a design space that maps semantic factors, such as descriptors, to continuous shapes representing the totality of possible designs.

[0237] The design space can be reversible, enabling the generation of visual representations that map design factors to continuous shapes representing potential product design geometry based on a dataset of user-provided information. This includes generating visual representations by mapping potential product design geometry to design factors.

[0238] Process 900 continues, dividing the visual representation into multiple segments based on design factor values ​​(908). As above, relative to... Figures 3 to 4 The model generator 126 is capable of dividing the design space into groups based on the values ​​of subjective factors.

[0239] In some implementations, segmenting a visual representation into multiple segments based on design factor values ​​includes dividing the visual representation into multiple segments based on design factor values, such that each segment of the visual representation shares design factor values ​​within a defined range of values. For example, data processor 124 can segment a design space into multiple segments based on a range of design factor values, such as a subjective rating of how comfortable a handbag design is perceived to be.

[0240] Process 900 continues by selecting segments containing visual representations of fewer than a threshold number of data points (910). As described above relative to... Figures 3 to 4 The data processor 124 is capable of identifying data segments based on metrics such as the number of data points, for example, a threshold number. For instance, the data processor 124 can identify a segment of a very "stylish" and very "practical" bag as having fewer than the threshold number of data points or failing to meet other metrics.

[0241] Process 900 continues, selecting the digital components from which information is solicited from the user (912). For example, task processor 122 or DCDS 110 can select the digital components from which information is solicited from the user. Task processor 122 can select or generate a task to solicit information from the user.

[0242] In some implementations, task processor 122 selects the format of the feedback solicited for the digital components from which information is requested from the user. For example, task processor 122 selects the format of the user's reaction to the design of a handbag.

[0243] In some implementations, selecting the format of the information solicitation includes choosing a specific feedback mechanism that is provided with dynamically changing digital components.

[0244] In some implementations, task processor 122 verifies information solicited by digital components based on specific attributes corresponding to underrepresented portions of the user population. For example, task processor 122 verifies that the information solicited for a task will be distributed based on the user's age attribute.

[0245] Process 900 continues, dynamically altering the presentation (914) of digital components that solicit information from the user about visual representation segments containing fewer than a threshold number of data points, based on the selected segments of the visual representation. For example, as described above relative to... Figures 3 to 4 The task processor 122 is capable of dynamically modifying existing tasks or generating new tasks. In one example, the task processor 122 can modify existing tasks to present new product designs to the user that were not previously generated or presented to the user.

[0246] In some implementations, dynamically changing the presentation of content items involves using machine learning or artificial intelligence techniques to specify information requested by the digital components. For example, data quality processor 120 is capable of using a machine learning model generated by model generator 126 to determine and specify the information requested by the task.

[0247] In some implementations, dynamically changing the presentation of digital components includes, based on a request for the digital components to be presented at the user device, determining that the user of the user device is in a first user cluster interested in a particular product design, and based on determining that the user of the user device is in the first user cluster interested in a particular product design, identifying user interface elements of the digital components, and changing the user interface elements that represent the presentation of the digital components. For example, data processor 124 is capable of determining that the user of user device 106 is in an interested user cluster, identifying user interface elements of a task, and changing the user interface elements to customize the task for a user who is already interested in a handbag design.

[0248] In some implementations, dynamically changing the presentation of a digital component includes, based on a request for the digital component to be presented on the user device, determining that the user of the user device is in a first user group interested in a particular product design, wherein the request for the digital component to be presented on the user device is based on information provided by the user indicating one or more attributes of the user, and based on determining that the user of the user device is in a first user group interested in a particular product design, identifying user interface elements of the digital component, and changing user interface elements of the digital component presentation.

[0249] In some implementations, process 900 includes constructing a behavioral model based on feedback information to predict user acceptance of potential product design geometry. For example, model generator 126 is capable of generating a behavioral model predicting user acceptance of a handbag product design. Modifications to design factors of a particular product design are based at least in part on the behavioral model.

[0250] Process 900 continues, distributing dynamically changed digital components for presentation at the user equipment (916). For example, DCDS 110 is able to distribute the task and any requested content as a response 114 to user equipment 106.

[0251] Process 900 continues, obtaining feedback information (918) from the user equipment and through a feedback mechanism regarding a segment of visual representation containing fewer than a threshold number of data points. For example, DCDS 110 is able to receive response data 116 from user equipment 106 and provide it to data processor 124. As described above relative to... Figures 3 to 4 The data quality processor 120 is capable of receiving feedback from the user regarding specific segments of the visual representation. For example, the data processor 124 and DCDS 110 are capable of receiving feedback from the user equipment 106 regarding design space segments having fewer than a threshold number of data points.

[0252] In some implementations, the request for a digital component to be presented on the user equipment indicates the user's user demographic information. Process 900 may also include, based on the request for a digital component to be presented on the user equipment, identifying the user of the user equipment within a first user group, such as a female user group in California.

[0253] In some implementations, process 900 may further include receiving from a second user device a request for a digital component to be presented on the second user device, the request indicating user demographic information of the user of the second user device. System 100 (e.g., data processor 124) can then determine, based on the request for the digital component to be presented on the second user device, that the user of the second user device is in the same first user group as the user of the user device. For example, system 100 (e.g., data processor 124) can determine that the user of the second device is a woman from California. In response to determining that the user of the second user device is in the same first user group as the user of the user device, system 100 can provide a modified product design instead of a specific product design. For example, due to the similarity between the user of the first user device and the user of the second user device, task processor 122 can provide the user of the second user device with a modified handbag design instead of the original handbag design.

[0254] Process 900 continues, modifying design factors of a specific product design, at least in part, based on feedback from the user, to create a modified product design (920). For example, model generator 126 is capable of modifying design factors of a handbag design, at least in part, based on feedback from the user, to create a modified handbag design.

[0255] As mentioned above Figures 3 to 4 The model generator 126 is capable of updating the design space and / or the behavioral model. For example, the model generator 126 can update the dataset by providing feedback information as input to the training system of the behavioral model or the design generator.

[0256] In some embodiments, process 900 includes identifying, from a plurality of existing product designs, the one closest to the modified product design in terms of having multiple or a maximum number of common design factor values. For example, data quality processor 120 is capable of identifying the existing product that is closest to the modified design. For example, system 100 is capable of modifying an existing product and its manufacturing method, rather than generating a completely new product. In some embodiments, system 100 is capable of providing the modified product design to an integrated manufacturing system. For example, data quality processor 120 is capable of providing the modified product design to a 3D printing system or an automated manufacturing system for immediate production.

[0257] Figure 10 This is a block diagram of an example computer system 1000 capable of performing the operations described above. System 1000 includes a processor 1010, a memory 1020, a storage device 1030, and an input / output device 1040. Each of components 1010, 1020, 1030, and 1040 can be interconnected, for example, using a system bus 1050. Processor 1010 is capable of processing instructions executed within system 1000. In one embodiment, processor 1010 is a single-threaded processor. In another embodiment, processor 1010 is a multi-threaded processor. Processor 1010 is capable of processing instructions stored in memory 1020 or storage device 1030.

[0258] The memory 1020 stores information within the system 1000. In one embodiment, the memory 1020 is a computer-readable medium. In one embodiment, the memory 1020 is a volatile memory cell. In another embodiment, the memory 1020 is a non-volatile memory cell.

[0259] Storage device 1030 provides high-capacity storage for system 1000. In one embodiment, storage device 1030 is a computer-readable medium. In various other embodiments, storage device 1030 may include, for example, a hard disk drive, an optical disk drive, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other high-capacity storage device.

[0260] Input / output device 1040 provides input / output operations for system 1000. In one embodiment, input / output device 1040 may include one or more network interface devices, such as Ethernet cards, serial communication devices (e.g., RS-232 ports), and / or wireless interface devices (e.g., 802.11 cards). In another embodiment, the input / output device may include a driver device configured to receive input data and send output data to other input / output devices (e.g., keyboards, printers, and display device 1060). However, other embodiments, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc., may also be used.

[0261] Despite Figure 10 An example processing system is described herein, but implementations of the subjects and functional operations described herein can be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more of them.

[0262] An electronic document (for simplicity, it will be referred to as a document) does not necessarily correspond to a file. A document can be stored as a part of a file that contains other documents, as a single file dedicated to the document in question, or as multiple collaborative files.

[0263] The embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a computer storage medium (or multiple media), for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions can be encoded on artificially generated propagated signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium can be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Furthermore, although the computer storage medium is not a propagated signal, it can be a source or destination of computer program instructions encoded in artificially generated propagated signals. The computer storage medium can also be or be included in one or more separate physical components or media (e.g., multiple CDs, discs, or other storage devices).

[0264] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0265] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including programmable processors, computers, systems-on-a-chip, or a combination thereof. The apparatus can include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus can also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0266] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment. A computer program may, but does not need to, correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple collaborative files (e.g., a file storing one or more modules, subroutines, or code sections). A computer program can be deployed to execute on a single computer or on multiple computers located in one place or distributed across multiple locations and interconnected by a communication network.

[0267] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processes and logic flows can also be executed by a dedicated logic circuit system, and the device can be implemented as a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0268] For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), and so on. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The processor and memory can be supplemented or incorporated by a dedicated logic circuit system.

[0269] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0270] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server, or middleware components, such as an application server, or front-end components, such as a client computer with a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0271] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship arises from computer programs running on their respective computers, and they have a client-server relationship. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from them). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0272] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in this specification in the context of independent embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination can be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0273] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0274] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for generating new product designs, comprising: Receive requests from the user's user equipment for digital components to be presented on the user equipment; Receive datasets containing user-provided information about specific product designs; Based on the dataset of information provided by the user, a visual representation is generated that maps design factors representing the subjective characteristics of the specific product design to a continuous shape representing the geometry of the potential product design. The visual representation is segmented into multiple segments based on the values ​​of the design factors; Select segments of the visual representation from the dataset that contain fewer than a threshold number of data points; Select the digital components from which information is solicited from the user; Based on selected segments of the visual representation, the presentation of the digital components is dynamically changed such that the dynamically changed digital components solicit information from the user about the segments of the visual representation that contain fewer than the threshold number of data points; Distribute the dynamically changed digital components to be presented at the user device; Feedback information is obtained from the user device and through a feedback mechanism regarding the segment of the visual representation containing fewer than the threshold number of data points; as well as The values ​​of given design factors for a particular product design are modified, at least in part, based on the feedback information obtained from the user, to create a modified product design.

2. The method according to claim 1, further comprising: The format for soliciting information is selected based on the digital components from which information is solicited from the user. as well as Based on the selected segment of the visual representation, verify the information solicited from the digital component that is used to solicit information from the user.

3. The method according to claim 2, wherein, Choosing the format for soliciting the information includes selecting a specific feedback mechanism that aligns with the dynamically changing digital components.

4. The method according to claim 1, further comprising: Based on the request for the digital components to be presented at the user equipment, the user of the user equipment is determined to be in a first user group. The request for the digital components to be presented on the user equipment indicates the user's user demographic information.

5. The method according to claim 4, further comprising: Receive a request from a second user equipment for a digital component to be presented at the second user equipment, the request indicating user demographic information of the user of the second user equipment; Based on the request for the digital components to be presented at the second user equipment, it is determined that the user of the second user equipment is in the same first user group as the user of the user equipment. as well as In response to determining that the user of the second user equipment is in the same first user group as the user of the user equipment, a modified product design is provided instead of the specific product design.

6. The method according to claim 1, wherein, Segmenting the visual representation into multiple segments based on the values ​​of the design factors includes: Based on the design factor values, the visual representation is divided into multiple segments, such that each segment of the visual representation shares design factor values ​​within a defined value range.

7. The method according to claim 1, wherein, Dynamically changing the presentation of the digital components includes: Based on the request for the digital component to be presented on the user equipment, the user of the user equipment is determined to be in a first user cluster interested in the particular product design, wherein the request for the digital component to be presented on the user equipment is based on information provided by the user to indicate one or more attributes of the user; Based on determining that the user of the user device is in a first user cluster interested in the specific product design, the user interface elements of the digital components are identified; and The user interface elements that change the presentation of the digital components.

8. The method according to claim 7, wherein, The user interface elements of the digital component are the visual themes of the digital component, and Changing the user interface elements includes modifying the visual theme of the digital components by modifying the color scheme and brand logo presented in the digital components.

9. The method according to claim 1, wherein, Dynamically changing the presentation of the digital components includes: Based on the request for the digital component to be presented on the user equipment, the user of the user equipment is determined to be in a first user group, wherein the request for the digital component to be presented on the user equipment includes information indicating one or more attributes of the user; Based on determining that the user of the user equipment is in the first user group, the user interface elements of the digital components are identified; and The user interface elements that change the presentation of the digital components.

10. The method according to claim 1, wherein, The visual representation that maps design factors to continuous shapes representing potential product design geometry is reversible, such that generating the visual representation that maps design factors to continuous shapes representing potential product design geometry based on a dataset of user-provided information includes generating the visual representation by mapping potential product design geometry to design factors.

11. The method according to claim 1, further comprising: Based on the modified product design and identifying the closest existing product design from a plurality of existing product designs, the existing product design having a plurality of design factor values ​​common to the modified product design.

12. The method of claim 1, further comprising providing the modified product design to the integrated manufacturing system.

13. The method according to claim 1, further comprising: Based on the feedback information, construct a behavioral model to predict the user's acceptance of potential product design geometry; as well as The design factors for modifying the specific product design are at least partially based on the behavioral model.

14. The method according to claim 1, wherein, The specific product design refers to the user interface design of the software application.

15. The method according to any one of claims 1-14, wherein, Dynamically changing the presentation of the digital component includes using machine learning or artificial intelligence techniques to specify the information to be requested by the digital component.

16. A system comprising: One or more processors; as well as One or more memory elements, said one or more memory elements including instructions that, when executed, cause said one or more processors to perform operations, said operations including: Receive requests from the user's user equipment for digital components to be presented on the user equipment; Receive datasets containing user-provided information about specific product designs; Based on the dataset of information provided by the user, a visual representation is generated that maps design factors representing the subjective characteristics of the specific product design to a continuous shape representing the geometry of the potential product design. The visual representation is segmented into multiple segments based on the values ​​of the design factors; Select segments of the visual representation from the dataset that contain fewer than a threshold number of data points; Select the digital components from which information is solicited from the user; Based on selected segments of the visual representation, the presentation of the digital components is dynamically changed such that the dynamically changed digital components solicit information from the user about the segments of the visual representation that contain fewer than the threshold number of data points; Distribute the dynamically changed digital components to be presented at the user device; Feedback information is obtained from the user device and through a feedback mechanism regarding the segment containing fewer than the threshold number of data points in the visual representation; and The values ​​of given design factors for a particular product design are modified, at least in part, based on the feedback information obtained from the user, to create a modified product design.

17. The system of claim 16, further comprising: Based on the request for the digital components to be presented at the user equipment, the user of the user equipment is determined to be in a first user group; as well as The request for the digital components to be presented on the user equipment indicates the user's user demographic information.

18. The system of claim 17, further comprising: Receive a request from a second user equipment for a digital component presented at the second user equipment, the request indicating user demographic information of the user of the second user equipment; Based on the request for the digital components to be presented at the second user equipment, it is determined that the user of the second user equipment is in the same first user group as the user of the user equipment. as well as In response to determining that the user of the second user equipment is in the same first user group as the user of the user equipment, a modified product design is provided instead of the specific product design.

19. A non-transitory computer storage medium encoded with instructions that, when executed by a distributed computing system, cause the distributed computing system to perform the operation of the method according to any one of claims 1-15.

Images